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ABSTRACT 

Technical  personnel  are  increasingly  being  required  to 
perform  vital  functions  as  proposal  evaluators  in  the  source 
selection  process  for  which  they  have  not  properly  been  trained, 
This  resource  effort  provides  a  comprehensive  system  for  source 
selection  using  price  and  other  factors  in  a  form  aimed  at  the 
technical  professionals  that  support  field  acquisition  activi- 
ties.  All  examples  selected  are  from  the  general  acquisition 
area  of  service  contracting.   The  system  consists  of  the  basic 
considerations  necessary  for  preparation  of  a  procurement 
request,  the  basic  elements  of  a  source  selection  plan,  selec- 
tion of  a  technical  evaluation  panel,  selection  of  evaluation 
factors  for  service  contractors,  preparation  of  negotiation 
objectives,  the  actual  conduct  of  evaluations  and  negotiations, 
and  debriefing  of  unsuccessful  offerors. 
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I.   INTRODUCTION 

A.   GENERAL 

In  the  past  ten  years  Navy  technical  personnal  have  been 
increasingly  required  to  be  contract  administrators  and 
proposal  evaluators.   This  fact  has  been  clearly  recognized 
at  major  technical  activities  such  as  the  Naval  Weapons  Center, 
China  Lake.   Senior  contract  specialist  positions  have  been 
established  in  several  of  the  major  technical  departments  to 
provide  liaison  with  the  acquisition  personnel  and  to  provide 
guidance  in  contract  administration  and  proposal  evaluation 
II].   However,  most  activities  supported  by  the  Navy  Field 
Acquisition  System  CNFAS)  cannot  afford  the  luxury  of  providing 
technical  personnel  with  contracting  advisors.   Nevertheless, 
th.e  technical  personnel  are  expected  to  perform  as  vital  mem- 
bers of  the  contracting  team. 

The  Department  of  Defense  does  provide  a  one  week  course 
on  "Defense  Contracts  Management  for  Technical  Personnel"  that 
covers  the  complete  acquisition  life  cycle   [2:1]  <»   Since  a 
significant  number  of  technicians  and  engineers  are  unable  to 
attend  this  course,  the  Navy  goes  one  step  further  and  provides 
a  one  week  course  for  "Contracting  Officer's  Technical  Repre- 
sentative CCOTR) . "   The  COTR  course  was  specifically  developed 
to  train  technical  personnel  in  contract  administration  and  is 
often  given  on  site.   However,  it  lightly  touches  the  area  of 
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proposal  evaluation   [3:11-5].   Interviews  with  NFAS  personnel 
confirm  that  a  void  exists  in  the  training  of  technical  per- 
sonnel in  proposal  evaluation  [4] . 

B.  STATEMENT  OF  THE  PROBLEM 

Poorly  planned,  defined,  conducted,  and  documented  pro- 
posal evaluations  have  contributed  to  the  following: 

a.  contractor  dissatisfaction  resulting  in  disputes, 
protests  and  claims  [5:3-49]. 

b.  qualified  contractors  deciding  not  to  submit  pro- 
posals or  submitting  inadequate  proposals  [5:3-49]. 

c.  long  procurement  administrative  leadtimes  [6:1-1]. 

d.  frustration  of  technical  personnel  [6:1]. 

e.  selection  of  unsatisfactory  contractors  (5:3-49], 

f.  difficulty  in  defending  selection  of  contractors  to 
the  General  Accounting  Office  (GAO)  [5:3-50], 

C,  RESEARCH  OBJECTIVE 

The  objective  of  this  research  effort  is  to  develop  a 
comprehensive  system  for  the  source  selection  of  service 
contractors  using  price  and  other  factors.   This  source 
selection  system  is  to  be  developed  with  the  technical  members 
of  the  evaluation  panel  in  mind. 

Service  contracts  were  selected  because  individuals  inter- 
viewed reported  service  contracts  as  the  most  common  area  to 
use  evaluation  factors  other  than  price.   It  is  important  to 
note  that  this  research  effort  does  not  address  modes  of  evalu- 
ation.  It  will  address  the  planning,  defining,  conducting, 
aggregating,  documenting,  and  use  of  evaluation  efforts  in  the 

source  selection  process. 
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D.  RESEARCH  QUESTION 

What  constitutes  a  thorough  and  impartial  source  selection 
system  using  price  and  other  factors  that  is  suitable  for  use 
as  a  standard  in  service  contracting  within  the  Navy  Field 
Acquisition  System? 

E.  SCOPE,  LIMITATIONS,  AND  ASSUMPTIONS 

1.  Scope 

A  broad  definition  of  service  contracts  is  used  to 
include  any  requirement  for  a  contractor's  time  and  effort 
rather  than  for  a  concrete  end  item.   Examples  of  contracting 
actions  that  typically  would  require  proposal  evaluation 
using  price  and  other  factors  are: 

a.  expert  and  consultant  services. 

b.  services  for  the  operation  of  Government  owned 
facilities  and  systems. 

c.  engineering  and  technical  services. 

d.  research  and  development  (R&D)  services. 

e.  automatic  data  processing  equipment  (ADPE) 
services. 

2 .  Limitations 

Contracts  for  services  that  are  acquired  using  small 
purchase  procedures  are  specifically  excluded.   Major  weapons 
systems  related  service  acquisitions  are  excluded  because  the 
proposal  evaluations  are  often  supported  by  complex,  computer 
models. 
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3 .   Assumptions 

It  is  assumed  that  the  reader  has  a  technical  back- 
ground and  only  a  minimal  understanding  of  Government 
contracting. 

F .   METHODOLOGY 

1.  Literature  Search 

A  comprehensive  search  was  conducted  of  the  literature 
base  for  applicable  studies  and  articles  in  the  areas  of 
source  selection  and  decision  and  policy  analysis.   Informa- 
tion was  obtained  primarily  from  the  Naval  Postgraduate  School 
library  and  the  Defense  Logistics  Information  Center,  Fort 
Lee,  VA. 

2.  Legal  Search 

The  primary  legal  search  method  was  a  detailed  review 
of  the  Quarterly  Digests  of  Unpublished  Decisions  of  the 
Comptroller  General  of  the  United  States  and  the  published 
Decisions  of  the  Comptroller  of  the  Unites  States.   A  second- 
ary method  was  a  computer  search  using  key  words  of  all  GAO 
decisions.   This  service  was  provided  by  the  Federal  Legal 
Information  Through  Electronics,  Denver,  CO.   The  legal 
search  concentrated  on  GAO  decisions  due  to  the  pre-award 
nature  of  proposal  evaluations.   GAO  decisions  are  issued  in 
response  to  protests  submitted  by  contractors  to  appeal  the 
actions  of  a  contracting  officer.   Very  seldom  do  contractors 
appeal  to  the  courts  concerning  proposal  evaluations,   A  com- 
puter search  of  federal  court  decisions  was  very  unproductive. 
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3.  Major  Activity  Source  Selection  Guides 

An  effort  was  made  to  collect  as  many  major  activity 
source  selection  guides,  handbooks  or  instructions  as  possible 
Examples  of  dociiments  collected  are: 

a.  U.  S.  Department  of  Energy,  "Procurement 
Regulation  Handbook,"  30  June  1979. 

b.  National  Aeronautics  and  Space  Administration, 
"Source  Evaluation  Board  Manual,"  change  2, 

25  January  1980. 

c.  U.  S.  Army  Material  Development  and  Readiness 
Command,  "Contract  Management  Guidance  for 
Technical  Personnel,"  advanced  copy. 

d.  "Source  Selection  Process  Handbook  for  the 
Air  Force  Space  Division,"  May  19  80. 

e.  Naval  Air  Development  Center,  Warminster, 
"The  Preparation  and  Processing  of  Purchase 
Requests , "  undated . 

4.  Personal  Lnverviews 

Finally,  visits  were  made  to  the  Naval  Supply  Center, 
Oakland  and  to  the  Naval  Regional  Contracting  Office,  Long 
Beach.   Personal  interviews  were  conducted  of  contract 
specialists,  management  personnel  and  legal  counsel.   The 
purpose  of  the  interviews  was  to  identify  and  evaluate  pro- 
posal evaluation  techniques  and  procedures  currently  being 
used. 
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II.   SOURCE  SELECTION  METHODS 

Nothing  in  contract  management  is  more  important  than 
an  informed  start.   Created  by  early  communications  between 
informed  personnel,  it  greatly  improves  the  probability  of 
an  uneventful  procurement.   The  usual  result  is  a  shortened 
cycle  for  the  award  of  a  contract  and  a  greater  potential 
for  satisfactory  performance  by  a  thoroughly  evaluated 
contractor.   [6:1] 

A.   BASIC  CONSIDERATIONS 

An  informed  start  begins  with  the  consideration  of  the 
full  spectrum  of  source  selection  methods  on  each  procure- 
ment.  The  most  basic  consideration  is  whether  the  procure- 
ment is  sole  source  or  competitive.   This  is  a  formal 
determination  that  must  be  approved  by  the  contracting 
officer.   Th.e  complexity  of  the  service  being  acquired  often 
dictates  a  need  for  more  complex  source  selection  procedures. 
The  time  available  for  specification  development  and  procure- 
ment action  may  limit  the  options  available.   The  availability 
of  personnel  that  might  be  part  of  the  Government  evaluation 
team  may  be  a  constraint.   A  very  real  limitation  may  be  the 
availability  of  sources  in  private  industry  that  are  capable 
of  providing  the  required  services.   Of  course,  a  vital  con- 
sideration is  the  estimated  dollar  value  of  the  procurement 
which  may  control  the  cost-benefit  of  investing  expensive 
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administrative  effort  into  source  selection.   This  list  of 
basic  considerations  is  not  comprehensive  and  is  provided  only 
as  a  start. 

B.   SOURCE  SELECTION  METHODS 
1.   Type  of  Contract 

In  acquiring  services  where  technical  or  engineering 
competency  is  not  the  controlling  factor,  source  selection 
in  terms  of  price  alone  may  be  sufficient.   Award  may  be  made 
to  the  lowest  responsive  and  responsible  offeror  and  a  pre- 
award  survey  may  be  relied  upon  to  assist  in  these  determina- 
tions   [2:18],   A  small  business  may  appeal  a  negative 
responsibility  determination  to  the  Small  Business  Administra- 
tion CSBA}  which  may  issue  a  Certificate  of  Competency.   This 
document  for  all  intent  and  purposes  is  a  conclusive  determina- 
tion of  positive  responsibility, 

Wh.en  to  use  evaluation  factors  in  service  contracting 
is  a  subject  of  much  heated  debate  and  contradictory  opinions. 
One  extreme  view  is  that  evaluation  factors  are  appropriate 
for  all  service  contracts  because  a  special  skill  can  always 
be  identified.   Examples  given  were  contracts  for  moving  and 
storage  [7]  and  for  mess  attendant  contracts  [8] .   A  much 
more  conservative  view  was  taken  by  legal  counselt  interviewed 
who  expressed  a  concern  for  the  use  of  evaluation  factors  on 
any  fixed  price  contracts  [9] .    GAO  has  provided  some  guidance 
on  these  questions  and  has  consistently  held  that  evaluation 
factors  may  be  used  in  source  selection  on  fixed  price  contracts 
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and  especially  on  cost  type  contracts.   Particularly,  when 
the  Government  "is  seeking  creativity  and  innovation  rather 
than  manpower  thus  making  factors  other  than  price  paramount" 
[10]. 

The  fact  is  that  a  service  requirement  may  lend  itself 
to  the  use  of  a  fixed  price  contract.   If  a  suitable  contrac- 
tor can  be  obtained  by  considering  pricing  alone,  then  consi- 
derable saving  can  be  obtained  in  contractor  proposal  costs, 
Government  administrative  cost  and  time.   For  that  reason 
the  Naval  Regional  Contracting  Office,  Long  Beach,  has  been 
instrumental  in  developing  a  new  contract  type  currently  being 
used  in  mess  attendant  contracts.   The  Fixed  Price,  Indefinite 
Quantity  with  Award  Amount  (FP (IQ) -W/AA)  type  contract  was 
developed  to  counter  the  negative  effects  of  heavy  competition 
without  the  use  of  evaluation  factors.   It  recognizes  the  fact 
that  the  bulk  of  the  contract  is  for  manpower  that  has  very 
little  variation  from  contractor  to  contractor.   The  contractor 
is  required  to  pay  employees  at  least  the  prevailing  wage  rate 
as  set  by  the  U.  S.  Department  of  Labor  (DOL)  and  seldom  does 
a  contractor  pay  more.   Therefore,  the  FPCIQ)-W/AA  type  con- 
tract establishes  a  fixed  service  rate  based  on  the  DOL  wage 
determination  and  the  contractor  invoices  for  each  service 
unit  delivered.   For  mess  attendant  contracts,  the  service 
unit  is  manhours  of  food  service  attendant  services  delivered. 

All  management  functions  are  grouped  under  a  separate 
item  entitled  "Management  and  Support  Price."   Award  is  made 
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to  the  responsive  and  responsible  bidder  that  submits  the 
lowest  bid  for  Management  and  Support  Price.   A  separate 
award  mechanism  is  used  to  motivate  the  contractor  to  provide 
quality  service.   At  no  time  are  evaluation  factors  used  in 
the  source  selection  process . 

However,  desire  to  use  a  fixed  price  contract  does 
not  mean  that  evaluation  factors  are  not  appropriate.   Educa- 
tional services  are  a  common  example.   Required  expertise  is 
clearly  identifiable  and  finite  end  products  can  be  estab- 
lished on  a  per  course  or  per  student  basis.   [7] 
2,   Qualification  Factors 

Another  method  of  source  selection  is  the  use  of 
special  standards  of  responsibility  or  qualification  factors. 
These  are  go-no  go  factors  such  as  the  geographical  location 
of  facilities,  degree  of  licensing  requirements  of  key  per- 
sonnel or  specific  experience  in  a  specialized  field  of  oper- 
ations  I2:17J.   Qualification  factors  should  be  objectively 
verifiable  and  judgment   should  not  be  a  significant 
requirement  [11] . 

Qualification  factors  may  be  used  alone  or  in  conjunc- 
tion with  evaluation  factors.   When  used  with  evaluation  fac- 
tors, they  should  be  set  out  separately  and  clearly  identified, 
Mixing  qualification  factors  with  evaluation  factors  may  delay 
the  start  of  procurement  action   [6:1-1].   The  use  of  qualifi- 
cation factors  in  this  manner  serves  to  notify  potential  con- 
tractors of  the  minimum  qualifications  and  resources  necessary 
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to  perform  the  proposed  work  of  a  given  procurement.  Offerors 
who  do  not  possess  these  requirements  should  not  be  encouraged 
to  incur  proposal  and  other  expenses  involved  in  a  competitive 
submission.  [12:4-2] 

3.   Evaluation  Factors 

In  the  evaluation- factor  method,  the  Government  activi- 
ties have  broad  latitude  in  determining  the  evaluation  factors 
and  the  proposal  evaluation  method  to  be  utilized.   The  only 
requirements  are  that  the  evaluation  method  provide  a  rational 
basis  for  source  selection  and  that  the  evaluation  itself  be 
conducted  in  good  faith  and  in  accordance  with  the  announced 
evaluation  factors  in  the  RFP.   Most  NFAS  activities  utilize 
numerical  point  ratings  in  an  attempt  to  quantify  what  is 
essentially  a  subjective  judgment.   However,  neither  GAO  nor 
procurement  regulations  require  that  proposals  be  evaluated 
on  thB  basis  of  numerical  scores.  [13] 

There  are  other  source  selection  methods  such  as  two 
step  formal  advertising  which  are  rarely  used  in  service 
contracting  [11] . 

C.   THE  SOURCE  SELECTION  PLAN  CSSP) 

A  formal  document  that  is  seldom  used  by  NFAS  activities, 
at  least  by  this  title,  but  which  is  commonly  used  in  major 
weapons  systems  acquisition,  is  the  Source  Selection  Plan 
CSSP)  [14:7].  In  short,  a  SSP  is  the  Government's  statement 
to  itself  as  to  how  it  intends  to  purchase  what  it  wants. 
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Most  NFAS  activities  do  not  require  such  a  document  because 
most  procurement  actions  do  not  have  a  complex  source  selec- 
tion procedure  and  the  benefits  do  not  warrant  the  added 
administrative  burden  [11] .    However,  several  individuals 
interviewed  agreed  that  the  use  of  a  SSP  would  be  warranted 
for  source  selection  on  high  dollar  value  service  contracts. 
The  extent  and  detail  of  the  SSP  would  increase  with  the 
complexity  and  potential  for  protest  [15] . 

The  SSP  is  a  major  component  of  the  total  acquisition 
plan  and  therefore  must  be  taJcen  into  consideration  during 
the  earliest  steps  of  the  acquisition  process.   In  service 
contracting /  the  other  major  component  is  the  statement  of 
work  CSOW) .   Ideally,  the  SOW  and  SSP  should  be  developed 
concurrently.   This  is  the  first  example  of  the  need  for  early 
information  flow  and  teamworJc  between  the  technical  and  con- 
tracting personnel.   The  need  for  teamwork  certainly  does  not 
end  here,  but  must  extend  through  the  entire  source  selection 
process.   Teamwork  is  the  most  vital  prerequisite  to  the 
success  of  any  acquisition.  [2:45] 
1.   Basic  Elements 

The  basic  elements  of  the  SSP  are:   [16:3-3] 
'       -  a  brief  description  of  the  acquisition. 

-  identification  of  the  innovation  or  expertise 
required . 

-  proposed  composition  of  the  technical  evaluation 
panel  CTEP)  . 

-  proposed  evaluation  factors  and  weights. 
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-  rating  or  scoring  technique. 

-  intent  to  use  preproposal  conference. 

-  summary  of  independent  cost  estimate  (ICE) , 

-  proposed  milestones. 

2.  Preproposal  Conference 

Preproposal  conferences  are  a  valuable  tool  and  they 
are  often  used  by  NFAS  activities  [11] .   They  serve  to  provide 
valuable  background  material  and  to  clarify  complex  SOW  and 
RFP  provisions.   SOW  and  RFP  inconsistencies  may  be  surfaced 
and  corrected  prior  to  proposal  submission.   The  result  can  be 
a  significant  savings  in  time  and  administrative  effort  as 
compared  to  individual  sequential  visits.   It  is  important 
that  all  Government  personnel  be  completely  briefed  prior  to 
the  conference  and  an  accurate  record  be  made  of  the  conference 
proceedings.  [16:3-19] 

3.  Independent  Cost  Estimate  (ICE) 

Careful  consideration  should  be  made  towards  the  devel- 
opment of  an  ICE.   There  is  a  definite  relationship  between 
the  Government's  ability  to  develop  an  accurate  ICE  and  the 
problems  which  will  be  encountered  later  in  the  source  selec- 
tion process.   An  ICE  is  important  because  it  will  enhance  the 
identification  of  potential  funding  problems  as  well  as  "buy- 
ins."   A  "buy- in"  is  where  the  offeror  purposely  underestimates 
on  a  cost  type  contract  and  plans  on  subsequent  cost  overruns 
should  he  be  awarded  the  contract  [16:3-8].   GAO  has  made 
attempts  to  require  that  an  ICE  be  developed  for  every 
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acquisition  that  may  require  a  detailed  cost/price  analysis 
[17] .   However,  an  ICE  is  expensive  and  time  consuming  to 
prepare  and  NFAS  activities  do  not  prepare  an  ICE  for  every 
acquisition   [7] . 

4.   Source  Selection  Plan  Milestones 

Reasonable  milestones  should  be  established  for  the 
following  critical  events: 

-  issuance  of  the  RFP. 

-  receipt  of  proposals . 

-  competitive  range  determination. 
~  receipt  of  Best  and  Final  CB&F) , 

-  contract  award. 

It  is  important  that  sufficient  time  be  given  to  both 
Government  personnel  and  contractor  personnel  for  each  step 
in  tke  contracting  process.   Undue  compression  of  the  con- 
tracting process  can  discourage  competition,  restrict  innova- 
tion, and  increase  errors  in  proposals  and  evaluations  [8]. 
Naval  Regional  Contracting  Office,  Long  Beach,  studies  have 
shown  that  the  most  time  consuming  effort  is  the  evaluation 
by  the  requiring  activity.   Also,  there  have  been  extensive 
delays  in  issuing  the  RFP  and  in  the  receipt  of  proposals  due 
to  RFP  amendments  [11] .   Again,  teamwork  is  the  key  to  meeting 
the  milestone  dates. 
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D.   STATEMENT  OF  WORK 

A  Statement  of  work  (SOW)  is  a  document  that  describes 
accurately  the  nonspecif ication  requirements  of  a  job  or 
work  effort  including  the  standards  used  to  determine  whether 
thB  requirements  have  been  met  [18:4]  .   The  function  of  a 
SOW  is  to  articulate,  as  straightfoirwardly  and  unambiguously 
as  possible,  the  requirements  that  will  lead  to  the  perform- 
ance of  work  that  achieves  job  or  work  effort  objectives 
[6:11-1].   This  is  a  tough  and  slippery  area  and  unless  one 
understands  the  nature  of  services  contracting  it  can  be 
troublesome  [6:111-11]. 

This  research  effort  does  not  include  the  development 
of  a  SOW.   Detailed  guidance  may  be  obtained  in  the  Department 
of  Defense  military  handbook  MIL-HDBK-245A(NAVY)  dated 
1  August  1978  entitled  "Preparation  of  Statement  of  Work 
CSOWl  . "   Also,  Office  of  Federal  Procurement  Policy  Pamphlet 
No.  4  entitled  "A  Guide  to  Writing  and  Administering  Perform- 
ance Statements  of  Work  for  Service  Contracts"  provides 
guidance . 

1,   Impact 

The  SOW  plays  a  critical  role  throughout  the  entire 
source  selection  process.   It  impacts  the  type  of  contract 
selected,  the  decision  of  offerors  to  propose,  the  credibility 
of  proposed  cost  or  price,  the  usefulness  of  the  SSP,  and  the 
level  of  resentment  of  unsuccessful  offerors  [6:11-2].   A 
clearly  written  SOW  will  allow  offerors  to  understand  the 
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Government  requirement  and  respond  with  detailed  plans  and 
approaches  for  accomplishment  of  the  tasks  specified.   It 
will  allow  offerors  to  make  reasonable  estimates  of  the 
resources  which  will  be  required  to  satisfy  the  Government's 
requirements.   A  poorly  written  SOW  is  misleading  and  presents 
a  poor  image  to  private  industry  [19:28],   The  RFP  may  be 
delayed  as  contracting  personnel  and  technical  personnel 
attempt  to  upgrade  the  SOW.   The  receipt  of  proposals  may  be 
delayed  due  to  extensive  offeror  questions  and  resulting 
amendments  to  the  RFP,   Finally,  the  evaluation  and  award  may 
be  delayed  due  to  wide  variations  in  proposals  and  lengthy 
negotiations  to  clarify  the  requirement.   Often,  a  contractor 
may  build  in  costly  contingency  allowances  if  a  clear  and 
definite  SOW  is  not  provided  [5:3-15].   A  well  defined  service 
can  be  acquired  using  a  fixed  price  or  incentive  type  contract, 
while  the  inadequately  defined  service  often  uses  time  and 
material  or  cost  type  contracts. 
2,   Matching  SOW  and  SSP 

It  is  extremely  important  that  the  SOW  and  SSP  be 
carefully  matched  to  each  other.   Normally,  the  first  two 
steps  taken  by  a  prospective  offeror  in  examining  an  RFP  is 
to  read  the  SOW  and  proposed  evaluation  factors  [6:IV-2]. 
The  most  common  mismatch  is  when  the  SOW  clearly  describes 
a  normal  work  effort  that  can  be  perfoirmed  by  personnel  trained 
to  one  level  of  expertise,  but  the  evaluation  factors  are 
established  for  a  worst  case  situation  and  indicate  a  much 
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higher  level  of  expertise.   This  has  become  a  very  coiranon 
area  for  protest  by  unsuccessful  offerors.   Although  this 
type  of  protest  is  very  difficult  for  an  offeror  to  win,  it 
does  cause  delays  and  considerable  Government  administrative 
effort  and  frustration  [15] . 

If  there  is  a  mismatch  between  the  SOW  and  the  evalu- 
ation factors,  an  experienced  contractor  will  direct  his 
proposal  towards  the  evaluation  factors  [11] .   If  the  SOW  is 
inherently  spongy,  the  contractor  may  direct  his  proposal 
specifically  at  the  evaluation  factors,  win  the  award  and  then 
take  off  in  pursuit  of  expanding  the  scope  of  work  during 
performance  or  cutting  bac]c  on  the  worJc  effort  [6:11-3], 
This  can  lead  to  considerable  follow  on  contract  administration 
problems. 

3 .   Writing  the  SOW  to  a  Standard 

A  standard  is  defined  as  an  acJcnowledged  measure  of 
comparison.   A  standard  in  order  to  be  useful  must  be  measur- 
able.  A  SOW  must  not  only  describe  a  particular  work  effort, 
but  must  aescribe  that  work  in  terms  of  standards  that  con- 
sistently establish,  a  particular  quality  level.   The  evalua- 
tion factors  are  also  selected,  defined,  and  weighted  to 
match-  the  same  quality  level.   Each  proposal  is  evaluated 
against  the  standards  of  the  SOW  and  not  against  each  other. 
This  provides  consistency,  objectivity  and  comparability 
between  proposals.   [18:4] 
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E.   TECHNICAL  EVALUATION  PANEL  (TEP) 
1.   Establishing  a  TEP 

Normally,  in  an  acquisition  where  source  selection  is 
based  on  an  evaluation  with  primary  emphasis  on  factors 
other  than  price,  a  technical  evaluation  panel  (TEP)  is  estab- 
lished to  perform  the  actual  evaluation  and  to  provide  a 
recommendation  to  the  contracting  officer.   The  panel  gener- 
ally consists  of  at  least  three  members  and  seldom  more  than 
five.   Some  major  activities  have  local  restrictions  against 
the  use  of  only  one  member  on  the  TEP  although  not  specifi- 
cally forbidden  by  procurement  regulations.   A  TEP  chief  is 
selected  to  chair  the  meetings  and  to  be  a  central  contact 
point.   The  vote  of  the  TEP  chief  should  be  equal  to  the  other 
panel  members  [11] .   Ideally,  the  TEP  should  be  identified  at 
least  by  organization  involved  and  position,  if  not  by  name, 
prior  to  submitting  the  procurement  request  (PR)  to  the  con- 
tracting activity  [2:7]  . 

In  selecting  TEP  members  consideration  should  be  given 
to  their  particular  areas  of  expertise  and  to  their  avail- 
ability to  perform  the  evaluation.   It  is  possible  that  some 
TEP  members  may  evaluate  proposals  in  all  areas  and  others  may 
evaluate  only  a  single  area  related  to  their  own  expertise. 
TEP  members  are  usually  from  the  requiring  activity  bui:  evalu- 
ators  may  be  used  from  any  Government  activity.   Nongovernment 
personnel  should  not  be  used  as  TEP  voting  members  but  may  be 
used  on  an  advisory/consultant  basis  if  they  are  acting  in  an 

official  capacity  or  under  contract  for  that  purpose.   If 
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consultants  are  to  be  used,  they  can  be  listed  in  tne  RFP  to 
surface  any  conflicts  of  interests  prior  to  initial  receipt 
of  proposals  [15] .    Consultants  should  be  allowed  access  only 
to  those  portions  of  the  proposal  that  are  necessary  to  enable 
them  to  give  specific  technical  advise  [16:3-5]. 
2.   Standards  of  Conduct 

TEP  members  for  major  weapons  systems  acquisition  are 
required  to  sign  a  statement  that  they  will  maJce  no  unauthor- 
ized disclosure  or  release  of  any  source  selection  informa- 
tion.  A  second  signature  is  required  on  a  statement  that  they 
have  no  personal  conflicts  of  interest  with  any  potential 
offeror.   Sample  statements  are  provided  as  Figures  1  and  2 
[14:31] . 

The  general  rules  of  conduct  are:   [14:31] 

-  direct  all  inquiries  pertaining  to  the  source 
selection  to  the  contracting  officer. 

-  do  not  permit  office  personnel  to  divulge  TEP 
member  activity  to  casual  callers. 

-  do  not  assume  that  a  non-participating  contractor 
can  be  told  anything  pertaining  to  the  source 
selection  activity. 

-  do  not  discuss  proposals  among  TEP  members  at 
social  events . 

-  do  not  accept  any  invitation  from  contractor 
personnel  to  participate  in  any  social  affair 
regardless  of  how  remote  it  may  be  to  the  TEP 
activity. 

do  not  discuss  TEP  sensitive  data  even  after  the 
announcement  of  a  winning  contractor. 
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Disclosure  of  Source  Selection  Information  Statement 


I  certify  that  I  will  make  no  unauthorized  disclosure 
or  release  of  any  source  selection  information.   This 
includes  the  contents  of  all  proposals  submitted  in 
response  to  (contract  number) ,  and  any  evaluation 
thereof.   I  further  certify  that  I  will  not  discuss 
with  any  unauthorized  person  the  panel's  finding  or 
decisions,  the  contractor's  approaches,  or  any  data 
generated  during  the  selection  process.   I  understand 
that  unauthorized  disclosure  or  release  of  any  source 
selection  information  may  subject  me  to  disciplinary 
or  adverse  administrative  action. 


Figure  1 


Conflict  of  xnterest  Statement 


I  certify  that  neither  I  nor  my  immediate  family,  to 
the  best  of  ray  knowledge,  possesses  any  financial 
interest  whatsoever  in  any  company,  parent  or  subsi- 
diary, which  is  proposing  or  quoting  on  or  is  any  way 
involved  in  the  acquisition  now  being  considered  by 
the  source  selection  board  of  which  I  am  a  member. 
Should  any  company  in  which  I  or  my  immediate  family 
have  a  financial  interest  submit  quotes  or  proposals 
to  my  board,  I  will  reveal  immediately  such  interest 
to  the  chairperson  of  the  source  selection  board. 


Figure  2 
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The  degree  of  technical  sophistication  as  well  as  the 
dollar  value  and  sensitivity  of  the  acquisition  are  among 
those  considerations  that  dictate  the  magnitude  of  the  TEP 
briefing  effort.   NFAS  activities  have  on  occasion  required 
TEP  members  to  sign  disclosure  and  conflict  of  interest  state- 
ments but  more  often  they  send  a  cover  letter  addressing  the 
subject  along  with  the  initial  proposals  [11] .   Whatever 
the  manner.  Information  security  can  be  improved  by  an  early 
briefing  of  the  TEP.   Another  important  factor  is  holding  to 
a  firm  schedule.   As  time  drags  on  there  is  a  greater  chance 
of  a  leak  [7]  . 

3.   Basic  Duties 

The  basic  duties  of  the  TEP  are:   [5:2-8] 

-  review  and  finalize  the  SOW. 

-  recommend  evaluation  criteria. 

-  recommend  sources  for  solicitation. 

-  evaluate  technical  proposals, 

-  participate  in  preproposal  conferences. 

-  prepare  technical  documentation  in  support 
of  contracting  officer. 

-  participate  in  contract  negotiations. 

-  participate  in  debriefing  unsuccessful  offerors. 

When  proposals  are  submitted  to  the  TEP  for  evalua- 
tion, the  members  should  be  advised  that  the  evaluation  must 
be  conducted  in  strict  accordance  with  the  evaluation  factors 
set  forth  in  the  RFP.   They  should  also  be  advised  that  the 
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evaluation  documentation  must  contain  a  reasonably  complete 
discussion  of  each  proposal  under  each  factor  [20:13-1]. 
The  members  should  be  made  aware  of  the  Freedom  of  Information 
Act  (FOIA)  and  that  the  evaluation  documentation  may  eventu- 
ally be  made  available  to  the  offerors.   In  many  ways  the 
FOIA  has  been  a  benefit  in  obtaining  better  evaluation 
reports  [11] .   This  subject  is  addressed  in  greater  detail  in 
Chapter  VI. 

The  TEP  evaluations  are  to  be  based  on  all  available 
information  including  proposals,  discussions,  reference  and 
other  appropriate  checks,  and  the  personal  knowledge  of  the 
individual  members  in  the  areas  of  their  expertise  [12:3-3], 
The  TEP  should  establish  an  internal  procedure  for  resolving 
individual  differences  of  opinion  and  negotiating  a  group 
result.   Differences  may  also  develop  between  the  TEP  and 
the  contracting  officer.   The  TEP  should  be  aware  that  the 
decisions  of  the  contracting  officer  are  final  and  the  TEP 
jnust  act  in  accordance  with  the  decision  [5:2-10].    Finally, 
th.e  TEP  should  keep  in  mind  that  the  Government  may  be  force 
to  bear  the  burden  of  a  defective  evaluation.   This  may  mean 
anyttiing  from  an  expression  of  displeasure  from  GAO  to  a 
directive  that  the  current  contract  be  cancelled  and  resoli- 
cited  [9] .    There  have  been  some  cases  where  the  TEP  evalu- 
ations  failed  to  give  a  fair  and  honest  consideration  to 
the  offeror's  proposal  and  the  offeror  was  entitled  to  recover 
proposal  preparation  costs  from  the  TEP  activity  [21] . 
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III.   MULTI-ATTRIBUTE  UTILITY  MEASUREMENT  (MAUM) 

Procuring  activities  may  exercise  considerable  discretion 
in  determining  the  particular  method  of  proposal  evaluation 
to  be  utilized.   The  only  requirements  are  that  the  method 
provide  a  rational  basis  for  source  selection  and  that  the 
evaluation  itself  be  conducted  in  good  faith  [13] .   However, 
there  are  entire  fields  of  study  directed  at  identifying  and 
improving  evaluation  techniques  and  procedures.   Decision 
analysis  is  a  discipline  for  systematic  evaluation  of  alter- 
native actions  in  order  to  make  a  choice  among  them  [22:vii]. 
Some  other  related  fields  are  systems  analysis  and  policy 
analysis  [23:4].   Extensive  research  and  field  testing  has 
been  conducted  to  identify  those  evaluation  techniques  and 
procedures  that  are  most  effective, 

A  sub-division  of  decision  analysis  that  has  been  gaining 

popularity  in  the  last  five  years  is  Multi-Attribute  Utility 

Measurement  (MAUM) .   MAUM  is  an  explicit  technique  that  is  a 

decision  making  tool.   The  MAUM  procedure  for  evaluating 

.multiple  attributes  of  various  alternatives  is  essentially 

the  same  process  as  NFAS  activities  currently  use  in  source 

selection.   Multiple  evaluation  factors  are  established  and 

weighted  and  each  proposal  is  scored  using  these  factors  by 

a  technical  panel  of  experts.   There  are  few  guidelines  on 

tha  proper  use  of  evaluation  factors  being  used  by  NFAS 

activities.   The  explicit  nature  of  MAUM  would  bring  a  sense 

of  direction  and  exactness  to  the  source  selection  process 

124]. 
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In  a  very  pure  definition,  MAUM  is  not  a  method  of  evalu- 
ation, but  rather  a  widely  applicable  method  for  organizing 
and  presenting  evaluation  information  [25:1].   After  the 
evaluation  factors  are  clearly  identified,  ranked  and 
weighted,  the  individual  evaluator  is  still  faced  with  the 
problem  of  how  to  evaluate  the  proposal  for  that  particular 
factor. 

A.  PURPOSE 

The  purpose  of  this  chapter  is  to  describe  a  version  of 
MAUM  that  is  simple  and  straightforward  so  that  any  TEP  that 
wishes  to  conduct  a  MAUM  evaluation  may  do  so  without  refer- 
ence to  any  other  source.   There  are  some  rather  sophisticated 
versions  of  MAUM  currently  being  used  by  major  weapons  systems 
activities  that  require  trained  analyst  and  computer  support. 
It  is  assumed  that  th.e  TEP  members  at  most  NFAS  activities 
are  not  trained  analysts  even  though  they  are  recognized 
experts  in  a  particular  field.   Therefore,  it  is  further 
assumed  that  the  TEP  would  reject  any  overly  complex  analyti- 
cal procedure. 

B.  SIMPLE  MULTI -ATTRIBUTE  RATING  TECHNIQUES  (SMART) 

The  Simple  Multi-Attribute  Rating  Technique  (SMART)  is  a 
version  of  MAUM  specifically  developed  for  the  non-professional 
analyst.   The  technique  was  developed  primarily  by  Dr.  Ward 
Edwards,  currently  the  Director,  Social  Science  Research 
Institute,  University  of  Southern  California.   It  has  been  used 
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primarily  in  the  evaluation  of  social  programs  such  as 
criminal  justice  programs  and  school  busing  plans.   Because 
of  the  nature  of  social  programs,  the  primary  input  to  their 
evaluation  is  the  judgment  of  experts  in  the  field.   In 
contrast,  the  physical  characteristics  of  the  hardware  is  a 
major  portion  of  a  weapons  system  evaluation.   Superior 
properties  can  be  determined  by  measurement  and  testing. 
Because  of  the  nature  of  service  contracting  efforts,  the 
primary  input  to  their  evaluation  is  also  the  judgment  of 
experts.   For  this  reason,  this  researcher  believes  that 
all  the  field  testing  of  SMART  on  evaluation  of  social  pro- 
grams establishes  its  validity  for  use  in  the  evaluation  of 
service  contractors . 

The  following  description  of  SMART  is  a  paraphrase  of 
written  material  provided  by  Dr.  Edwards  as  supplemented  by 
a  personal  interview.   The  primary  difference  is  that  the 
nomenclature  has  been  converted  from  analytical  language  to 
procurement  language. 

The  basic  assumptions  of  SMART  are   [25: 1-3 J 

-  requests  for  services  normally  have  multiple 
goals  which  are  not  equally  important. 

-  judgments  are  inevitably  a  part  of  any  evaluation. 

-  judgments  of  magnitude  are  best  when  expressed 
numerically. 

-  evaluations  are  directly  relevant  to  the  source 
selection  decision. 
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Because  a  source  selection  must  be  defensible,  evaluators 
may  be  hesitant  to  use  judgment  to  arrive  at  a  recommendation. 
However,  unless  the  judgment  of  the  TEP  is  clearly  without  a 
reasonable  basis,  or  there  is  an  abuse  of  discretion,  or  a 
violation  of  procurement  statutes,  the  judgment  of  the  TEP 
will  not  be  disturbed  [26].   Nevertheless,  if  the  TEP  uses  an 
intuitive  method  of  scoring  an  evaluation  factor  when  indus- 
try standards  are  available,  they  could  expect  strong  criti- 
cism because  it  is  expected  they  will  use  hard  facts  and 
data  wh-en  available  [27:11]. 

Sometimes  expert  evaluators  are  reluctant  to  express 
their  judgments  in  numerical  form.   However,  Dr.  Edwards 
states,  "...th.e  evidence  abundantly  indicates  that  numerically 
expressed  judgments  are  far  more  accurate  and  far  more  useful 
th.an  human  judgment  expressed  in  the  more  familiar  quantita- 
tive verbal  forms."  [28:181]     A  sound  narrative  is  still 
required,  but  the  numbers  lend  themselves  to  convenient  manip- 
ulation, aggregation,  and  trade  off  evaluation. 

Dr.  Edwards  suggests  three  steps  to  be  used  to  increase 

the  liJcelihood  that  the  evaluation  will  be  used  in  the 

decision  [25:1-10], 

~   involve  the  contracting  officer  or  source 

selection  official  in  the  evaluation  process. 

-  make  the  evaluation  directly  relevant  to  the 
source  selection  decision. 

—  make  the  product  of  the  evaluation  readable  and 
easy  to  use. 
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C.   ELEMENTS  OF  SMART  SYSTEM 

The  SMART  system  is  characterized  by  a  ten  step  procedure. 
The  basic  technique  does  not  change,  only  the  identity  and 
expertise  of  the  typical  evaluators. 

1,  Step  One;   Identify  the  person,  persons,  organization, 
or  organizations  whose  values  are  to  be  maximized. 

Both  persons  and  organizations  will  usually  be  rele- 
vant generic  "stakeholders."  [28:189]    A  typical  series  of 
questions  to  ask  are: 

Who  are  the  end  users  of  the  service? 

Who  are  the  potential  contractors? 

-  Who  is  the  contracting  officer? 

-  What  is  che  potential  for  SBA,  Congressional 
or  other  high  level  involvement? 

2,  Step  Two:   What  is  the  purpose  of  the  evaluation? 
This  is  more  than  simply  source  selection.   The  ulti- 
mate purpose  is  to  provide  a  service  in  order  to  fill  an 
identified  requirement. 

3,  Step  Three:   Identify  the  entities  to  be  evaluated. 

A  feature  of  the  evaluation  process  for  source  selec- 
tion is  that  the  evaluation  factors  must  be  set  forth  in  the 
RFP  and,  therefore,  before  receipt  of  proposals.   Although 
not  normally  required  by  the  SMART  technique,  it  does  improve 
the  objectivity  of  the  evaluation  [24] .   However,  it  does 
require  substantial  prior  knowledge  of  potential  contractors 
and  theix  essential  characteristics. 
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It  is  important  to  decide  whether  the  evaluation  will 
be  based  solely  on  the  technical  content  of  the  proposals  or 
be  expanded  to  include  the  contractors  themselves.   The 
evaluation  of  a  contractor  requires  the  use  of  responsibility 
type  factors  such  as  "experience"  and  "facilities."   It  is 
improper  to  use  only  responsibility  type  factors.   However, 
it  is  proper  to  use  responsibility  type  factors  in  conjunction 
with  evaluation  factors  bearing  on  technical  approach  when 
used  to  make  relative  assessments  of  the  merits  of  individual 
proposals.   A  separate  responsibility  finding  is  made  on  the 
contractor  in  line  for  award  after  the  proposal  evaluation 
has  been  completed  [10] . 

4.   Step  Four:   Identify  the  relevant  evaluation  factors. 

The  motto  of  decision  analysis  is  "divide  and  conquer." 
[25:31]    This  means  that  the  evaluation  problem  should  be 
divided  into  separate  elements  and  then  recombined  by  means 
of  an  appropriate  formal  aggregation  rule.   In  addition,  the 
judgmental  tasks  should  be  partitioned  to  fit  the  expertise  of 
the  evaluators. 

This  evaluation  factor  structuring  task  is  probably 

the  most  important,  difficult  and  creative  aspect  of  SMART. 

The  procedure  recommended  by  Dr.  Edwards  is  to  develop  an 

evaluation  factor  tree.   A  hierarchical  structure  of  evaluation 

factors  with  more  abstract  and  unmeasurable  factors  at  the  top 

and  better  defined  and  more  measurable  sub-factors  at  the 

bottom  I28:191J.   CSee  figure  3.)   The  advantages  of  the  tree 

structure  are:   [25:2-8] 
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step  Four 
Identify  the  relevant  evaluation  factors 
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it  presents  factors  in  a  more  orderly  structure, 
thereby  helping  though  about  the  problem. 

it  can  help  reduce  the  number  of  judgments 
required. 

it  permits  subaggregation. 
Experience  shows  that  there  is  a  tendency  for  individ- 
uals to  add  factors  while  seldom  will  anyone  eliminate  a 
factor.   The  result  often  is  too  many  evaluation  factors. 
Ideally,  only  four  or  five  factors  are  enough.   However,  since 
many  more  factors  than  that  must  usually  be  considered. 
Dr.  Edwards  recommends  that  eight  are  enough  and  fifteen  are 
definitely  too  many.   The  reasons  why  the  number  of  evaluation 
factors  should  be  kept  small  are:   [28:19] 

-  a  large  number  of  factors  dilutes  the 
importance  of  an  important  factor  relative 
to  another . 

-  more  work  is  required  to  gather  necessary 
information. 

-  more  difficult  to  maintain  independence 
between  factors . 

Dr.  Edwards  recommends  several  methods  for  reducing 

the  number  of  factors:    [2  8:192] 

-  assemble  the  TEP  face-to-face  to  improve 
communications . 

~   look  for  factors  that  are  simply  the  relabeling 
of  other  factors . 

-  look  for  factors  that  have  a  high  environmental 
correlation 

-  eliminate  the  factors  that  are  not  important 
enough  to  influence  the  decision. 
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An  example  of  environmental  correlation  would  be  in 
the  evaluation  of  senior  academic  personnel  to  provide 
training  services.   There  would  be  a  high  correlation  between 
the  degree  status  and  the  number  of  published  writings.   One 
should  either  combine  the  factors  and  evaluate  together  or 
select  one  factor  to  use  as  a  proxy  for  the  other  [24] . 

5.  Step  Five;   Rank  the  factors  in  relative  order  of 
importance. 

For  evaluation  factors  arranged  in  a  tree,  factors 
beneath  each  node  in  the  tree  are  ranked  separately.   This 
means  the  major  factors  would  be  ranked  with  each  other  and 
the  sub-factors  would  be  ranked  separately  under  each  major 
factor.   CSee  figure  4.) 

6.  Step  Six;   Assign  weights  to  each  evaluation  factor. 
Weights  are  useful  because  all  the  factors  in  the 

evaluation  tree  are  not  of  equal  importance.   VJeights  serve 
as  devices  that  represent  the  relative  importance  of  each 
factor  to  all  others.   They  establish  an  exchange  rate  among 
evaluation  factors.   The  SMART  technique  allows  the  use  of 
several  weigh.ting  procedures,  but  Dr.  Edwards  recommends  the 
"ratio  method."  124 J 

To  assign  weights  using  the  ratio  method,  start  with 
the  least  important  major  factor  and  assign  a  weight  of  10. 
Then  compare  this  factor  with  the  next  most  important  factor 
and  assign  a  weight  that  depicts  how  much  more  important  that 
factor  is  relative  to  the  least  important  factor.   For  example, 
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step  Five 
Rank  the  factors  in  relative  order  of  importance 
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in  figure  4,  factor  B  is  twice  as  important  as  factor  A  and 
therefore  receives  a  weight  of  20.   In  working  up  the  list 
of  major  factors,  the  TEP  must  consider  not  only  the  least 
important  factor,  but  all  other  factors  that  have  already 
been  assigned  weights.   Constant  adjustments  of  weights  are 
to  be  expected.   After  the  major  factors  are  completed, 
each  individual  group  of  sub-factors  is  assigned  weights 
using  exactly  the  same  ratio  method  starting  with  a  weight 
of  IQ.   Ties  are  permitted  but  not  encouraged  because  they 
complicate  full  disclosure  in  the  RFP  [25:4-6]. 

7.   Step  Seven;   Normalize  the  weights  and  calculate  the 
final  weight  assignments. 

Normalization  is  a  simple  mathematical  technique  used 
to  make  all  the  factor  weights  add  up  to  100  and  for  each 
group  of  sub-factors  to  add  up  to  the  weight  assigned  the 
major  factor.   The  weights  assigned  the  major  factors  are 
summed.   The  weight  assigned  each  major  factor  is  divided  by 
th-e  sum  of  the  major  factors  and  multiplied  by  100  to  give  the 
final  weight.   For  example,  in  figure  5,  the  final  weight  for 
factor  C  is  calculated  dividing  the  original  weight  of  factor 
C  C5Q1  by  the  sum  of  all  original  weights  (115)  and  multiplying 
by  100  to  get  43.478.   Obviously,  any  mathematical  technique 
that  starts  with  judgmental  inputs  is  not  accurate  to  three 
decimal  places.   Therefore,  each  TEP  should  discuss  this  issue 
and  come  to  a  consensus  in  order  to  maintain  comparability 
between  the  evaluations  of  each  panel  member.   For  this  example, 
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step  Six 
Assign  weights  to  each  evaluation  factor 
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the  final  weight  of  factor  C  is  rounded  to  4  3.5.   Each  group 
of  sub-factors  is  normalized  in  exactly  the  same  manner, 
except  that  instead  of  multiplying  by  100,  multiply  by  the 
major  factor  weight.  (See  figure  6.) 

A  side  benefit  of  this  step  is  that  the  folly  of 
including  too  many  evaluation  factors  becomes  apparent.   If 
a  factor  has  a  final  weight  less  than  or  approaching  1,0,  it 
may  be  trivial.   One  should  consider  combining  this  factor 
with  another  factor  or  deleting  the  factor  altogether  [29:11], 
8.   Step  Eight;   Scoring  and  rating  the  proposals. 

A  score  or  a  rate  is  an  assessment  of  how  desirable 
a  particular  proposal  is  with  respect  to  a  particular  factor. 
Prior  to  the  receipt  of  proposals,  a  scoring  or  rating 
method  should  be  established  for  all  judgmental  factors. 
This  will  aid  each  TEP  member  in  scoring  and  increase  the 
comparability  between  their  scores.   A  common  technique  is  to 
establish,  a  scale  that  relates  adjective  statements  with 
numerical  values.   This  method  is  typically  used  in  personnel 
evaluations.  130:12] 

In  developing  a  scoring  or  rating  scale,  several 
points  should  be  considered:   [25:5-5] 

-  evaluators  are  reluctant  to  approach  the 
ejitremes,  especially  the  lower  extremes. 

the  scale  should  serve  to  differentiate  one 
proposal  from  another  and  therefore  should 
be  spread  out. 

-  the  scale  should  allow  for  the  evaluation 
of  proposals  outside  the  expected  range. 
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step  Seven 
Normalize  the  weights  and  calculate  the  final  assignment 
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There  is  a  close  relationship  between  the  scoring  or 
rating  scale  and  the  weights  assigned  in  Step  Six.   Separate 
scales  can  be  developed  for  each  evaluation  factor  based  on 
the  expected  range  of  proposals.   Within  the  expected  range, 
an  acceptable  zone  is  identified  bounded  by  the  minimum  accep- 
table and  maximum  acceptable.   For  ease  of  computations  and 
with  little  loss  of  accuracy  the  scale  within  the  acceptable 
zone  is  normally  assumed  to  be  linear.   The  weights  assigned 
are  good  only  for  proposals  in  this  acceptable  zone  [25:5-12]. 
In  the  evaluation  of  social  programs  the  weights  may  be 
adjusted  if  the  alternatives  are  different  than  original 
expectations.   However,  because  DAR  does  not  allow  major 
changes  in  weights  after  the  receipt  of  proposals,  mechanisms 
should  be  designed  into  the  scoring  or  rating  scale 
in  order  to  consider  proposals  outside  the  original  acceptable 
zone. 

From  a  practical  standpoint,  a  universal  scoring  or 
rating  system  can  be  used  instead  of  developing  a  scale  for 
each  factor.  Alterations  are  then  limited  to  as jus ting  the 
we^ights.  An  example  of  a  universal  scoring  or  rating  scale 
is  provided  below.  Notice  that  the  evaluator  does  not  have  to 
know  the  weight  assigned  a  particular  factor  in  order  to  score 
that  factor. 

1.0   Exceeds  specified  performance  or  capability  and 
excess  is  useful  (not  gold  plating) . 
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.9   Outstanding  -  a  definite  strength  that  meets  all 
the  requirements  of  the  RFP  with  a  high  probability  of  success, 

. 8   Above  average  -  a  minor  asset  but  with  room  for 
improvement. 

,7  Average  -  neither  a  strength  or  weakness  with 
significant  room  for  improvement. 

. 6  Below  average  ~  a  minor  deficiency  but  acceptable 
without  correction. 

Q.O   Unsatisfactory  -  fails  to  meet  the  standards  of 
the  RFP  and  is  not  correctable  without  a  major  rewrite  of  the 
proposal. 

This  universal  scoring  or  rating  scale  has  a  mechanism 
for  evaluating  excess  above  the  acceptable  range  and  a  mechan- 
ism for  making  unacceptable  proposals  stand  out  numerically. 
It  allows  excess  to  be  considered  which  encourages  innovation 
and  alternative  proposals.   Excess  is  set  apart  and  allowed  to 
enter  into  the  computations  only  if  the  excess  is  useful. 
The  w^ide  spread  between  Weak  and  Unsatisfactory  is  important 
because  it  forces  the  evaluators  to  address  the  critical 
question  whether  the  deficiency  is  capable  of  being  corrected. 
If  a  major  factor  is  unacceptable,  the  entire  proposal  is 
unacceptable.   Numerically,  a  zero  score  on  a  major  factor 
would  clearly  set  a  proposal  apart  from  other  proposals  that 
had  scored  at  least  fifty  percent  in  all  major  factors. 

The  mathematics  of  using  this  scale  are  illustrated 
by  figure  7. 
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step  Eight 
Scoring  and  rating  the  proposals 
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9.   Step  Nine;   Aggregate, 

The  aggregation  rule  is  very  simple  and  is  illustrated 
by  figure  8.   The  larger  the  final  numerical  score,  the  better 
the  proposal.   Normally,  the  proposal  with  the  largest  numeri- 
cal value  would  be  considered  the  best  proposal  [25:6-2]. 
10..  Step  Ten;   Decide. 

It  is  important  to  realize  that  SMART  is  only  a 
decision  making  tool  and  the  results  are  not  the  decision  [33] . 
The  results  must  be  presented  to  the  contracting  officer  or 
the  source  selection  official  in  a  manner  that  objectively 
presents  and  supports  the  recommendation  of  the  TEP.   Important 
trade-offs  and  options  should  be  identified.   In  complex,  high 
dollar  value  procurements,  a  sensitivity  analysis  may  require 
computational  support.   Dr.  Edwards  has  developed  programs  for 
th.e  TI-.59  and  HP-41C  hand  held  programmable  calculators  which 
are  available  upon  request. 
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step  Nine 
Aggregate 
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IV.   GOVERNMENT  CONSTRAINTS 

Although  procuring  activities  have  broad  latitude  in  the 
source  selection  process,  there  are  a  considerable  number  of 
constraints  under  which  they  must  operate.   Surprisingly,  the 
major  body  of  these  constraints  does  not  come  from  the  Defense 
Acquisition  Regulations,  but  from  the  decisions  of  the  Comp- 
troller General  of  the  United  States.   These  decisions  are  a 
result  of  protests  filed  by  contractors  or  the  result  of  formal 
requests  by  administrative  agencies  for  a  GAO  decision.   Once 
the  decision  is  made  and  issued,  it  is  binding  on  the  procur- 
ing activity.   Decisions  of  general  interest  are  published 
monthly  in  pamphlets  and  annually  in  volumes.   [31:13] 

A.   SUMMARY  OF  THE  RULES 

The  following  is  a  summary  of  the  rules  as  extracted  from 
the  GAO  decisions  and  other  literature  by  the  researcher  con- 
cerning the  source  selection  process.   It  is  by  no  means 
complete  and  any  specific  question  should  be  referred  to  legal 
counsel. 

1.   Procuring  activities  have  broad  latitude  in  determining 

the  particular  method  of  proposal  evaluation  [13] . 

The  method  must  provide  a  rational  basis  for 
source  selection. 

-  The  evaluation  must  be  conducted  in  good  faith. 

-  Numerical  scoring  is  not  required. 
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GAO  will  not  disturb  unless  shown  to  be  arbitrary 
or  contrary  to  statutes  or  regulations. 

2.  All  offerors  in  a  negotiated  procurement  must  be 
informed  of  the  major  evaluation  factors  and  their  relative 
importance. 

-  The  weights  of  the  evaluation  factors  shall  not 
be  disclosed  [32] . 

-  If  one  factor  is  assigned  a  predominant  value, 

its  greater  relative  importance  shall  be  indicated 
by  appropriate  language  [33] . 

If  no  indication  of  the  relative  importance  of 
factors  or  sub-factors  is  given,  offerors  may 
assiime  equal  importance  [34]. 

-  Any  minimum  standards  applicable  to  a  factor  must 
be  revealed  [35:22], 

-  Various  aspects  of  factors  need  not  be  explicitly 
identified  provided  such  aspects  are  logically 
related  to  the  stated  factor  [36] . 

Relative  importance  of  "definitive"  or  descriptive 
sub-factors  need  not  be  revealed. 

Relative  importance  of  sub-factors  that  are 
essential  characteristics  or  measures  of  perform- 
ance shall  be  revealed  [37] . 

3.  GAO  will  not  become  involved  in  appraising  the  quali- 
fications of  evaluators  [3  8] . 

-  In  allocations  of  bias,  the  protester  has  the 
burden  of  proof  [39] . 

-  Th-e  evaluation  panel  may  be  one  individual  or 
a  team  [40: 3] . 

-  All  evaluators  are  not  required  to  evaluate 
revised  proposals  [41] . 

4.  Evaluation  factors  may  be  used  in  the  negotiation  of 
fixed  price  contracts,  particularly  if  the  desired  product  is 
creativity  and  innovation  rather  than  manpower  [10] . 
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Proposal  evaluation  should  not  be  considered 
responsibility  findings  [10] . 

5.  Evaluation  of  detailed  resumes  and  firm  commitment 

of  key  personnel  do  not  create  a  prohibited  personal  services 
contract  [42] . 

-  If  offerors  are  contractually  committed  to  provide 
listed  k&Y   personnel,  letters  of  commitment  become  excess  [43] , 

-  Requirement  that  ]<ey  personnel  be  presently 
employed  with  offeror  is  unduly  restrictive  [44] . 

6.  It  is  fundamental  in  the  award  of  a  cost  type  contract 

that  proposed  cost  be  analyzed  in  terms  of  cost  realism  [45] . 

Merely  comparing  the  estimated  costs  proposed  by 
offerors  is  inadequate  [46] . 

An  independent  cost  projection  should  be  used  in 
the  evaluation  of  the  reasonableness  of  proposed 
cost  [17]. 

-  It  is  permissible  to  reveal  the  government  esti- 
mate and  to  use  it  in  negotiations  [47]. 

GAO  has  cautioned  against  undue  reliance  on  the 
government  cost  estimate  [47] . 

-  Phase-in  cost  factors  may  be  used  only  if  set 
forth  in  the  RFP  [48] . 

-  If  regulations  require  the  use  of  certain  factors, 
they  must  be  used  even  if  not  disclosed  in  RFP. 
Examples  are  transportation  costs  and  freight 
costs.  [49] . 

-  Intent  to  use  Government  property  is  3.   required 
factor  if  there  are  indications  that  one  or  some, 
but  not  all  offerors  will  require  the  property 
150]. 

-  Evaluated  cost  rather  than  proposed  cost  provides 
a  more  sound  basis  for  determining  the  most 
advantageous  proposal  [51] . 
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7.  It  is  not  the  function  of  GAO  to  make  determinations 
as  to  the  acceptability  or  relative  merits  of  technical 
proposals  [26] . 

-  The  TEP  may  consider  its  own  experience  with  the 
offeror  in  evaluating  the  proposals  [52] . 

The  Government  may  use  its  own  reasonable  esti- 
mate of  manhours  to  measure  the  offeror's  under- 
standing of  requirements  [53] . 

-  In  an  unrestricted  procurement,  it  is  improper 
to  evaluate  small  business  proposals  differently 
[54]  . 

Evaluation  of  prior  experience/past  performance 
i.s  not  improper  or  discriminatory  with  respect 
to  small  business  [10] . 

-  The  contracting  activity  must  bear  the  burden  of 
any  difficulties  resulting  from  a  defective 
evaluation  [26] . 

-  If  an  evaluation  is  found  to  be  arbitrary  and 
capricious,  the  low  offeror  may  be  entitled  to 
proposal  preparation  cost  [21] . 

8.  Proposals  that  do  not  provide  sufficient  detail  to 
permit  technical  evaluation  without  complete  rewrite  may  be 
rejected  as  technically  unacceptable  [55] , 

9.  Award  may  be  based  on  initial  proposals:   [40:4] 

if  urgent  delivery  requirements  will  not  permit 
discussions. 

-  if  the  existence  of  adequate  competition  or 
accurate  cost  experience  can  be  clearly 
demonstrated . 

-  if  acceptance  of  the  most  favorable  initial 
proposal  would  result  in  a  fair  and  reasonable 
price  to  the  Government. 

10.   Unless  the  award  is  to  be  made  on  a  proposal  as 

initially  submitted,  written  or  oral  discussions  must  be  held 

with  all  responsible  offerors  in  the  competitive  range  [40:5]. 
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The  use  of  pre-determined  cut-off  scores  to 
establish  the  competitive  range  is  not  in 
accordance  with  sound  procurement  practice  [56], 

Proposals  may  be  found  outside  the  competitive 
range  on  the  basis  of  cost  or  technical  factors 
without  consideration  of  the  other,  however,  a 
technically  acceptable  proposal  may  not  be 
found  outside  the  competitive  range  without 
consideration  of  cost  [40:5], 

-  Once  a  proposal  is  included  in  the  competitive 
range,  it  cannot  be  excluded  without  an  oppor- 
tunity to  submit  a  revised  proposal  [40:5], 

-  An  unsuccessful  offeror  is  to  be  advised  that 
he  is  no  longer  in  contention  unless  award  is 
to  be  made  in  a  few  days  [40:9]. 

11.  The  Government  must  usually  furnish  information  to 
offerors  as  to  the  areas  in  which  their  proposals  are 
deficient. 

-  It  is  not  required  to  discuss  inad-equate  areas 
of  a  proposal  to  the  extent  technical  trans- 
fusion or  technical  levelling  would  take  place 
157]. 

The  offeror  has  a  right  to  know  that  discussions 
are  being  conducted.  Best  and  Final  (B&F)  offers 
are  required,  and  the  deadline  for  submission 
C40:7]  . 

-  A  second  B&F  is  permitted  where  justifiable 
I4Q:7J . 

-  Technical  scores  may  be  reduced  after  second 
evaluation  [36] . 

12.  Where  selection  officials  have  determined  proposals 
to  be  technically  equal,  cost  can  become  the  determining  fac- 
tor in  award  [58] . 

-  A  determination  of  technical  equality  must  be 
made.   Simply  awarding  to  the  lowest  qualified 
offeror  fails  to  follow  the  announced  evalua- 
tion scheme.   [59] 
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A  factual  explanation  why  proposals  are  essentially- 
equal  is  required.   A  bare  conclusionary  statement 
is  not  sufficient.   [60] 

-   Whether  a  given  point  spread  indicates  a  signifi- 
cant superiority  of  one  proposal  over  another 
depends  on  the  facts  and  circumstances  of  each 
procurement  [61] . 

13.  The  contracting  officer  or  source  selection  official 
is  not  bound  by  the  recommendations  of  the  TEP  provided  such 
selection  is  reasonable  and  consistent  with  the  evaluation 
criteria  [62] . 

14.  Protests  which  are  based  upon  alleged  improprieties  in 
the  evaluation  scheme  and  which  are  apparent  prior  to  the 
closing  date  for  receipt  of  initial  proposals  must  be  filed 
prior  to  the  closing  date  for  receipt  of  initial  proposals. 
Protests  alleging  improprieties  in  the  evaluation  of  proposals 
must  be  filed  not  later  than  ten  days  after  the  basis  for  pro- 
teat  becomes  known  or  should  have  become  know,  whichever  is 
earlier.   [35:22] 

B.   PROTECTIVE  FUNCTION  OF  SBA 

As  noted  in  Chapter  II,  the  SBA  has  a  role  in  the  deter- 
mination cf  responsibility  with  respect  to  small  business 
offerors.   The  SBA  has  statutory  authority  to  certify  the 
competency  of  any  small  business  concern  as  to  all  elements 
of  responsibility,  including,  but  not  limited  to,  capability, 
competency,  capacity,  credit,  integrity,  perseverance,  and 
tenacity.   The  Defense  Acquisition  Regulation  (DAR)  1-705.4 
states  that  the  contracting  officer  shall  accept  as  conclusive 
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SBA  certificates  of  competency  unless  the  contracting  officer 
has  substantial  doubt  as  to  the  small  business  ability  to 
perform.   In  this  case  the  contracting  officer  must  follow  a 
review  and  appeal  procedure  set  forth  in  DAR  1-705. 4(f). 

A  few  contracting  officers  have  attempted  to  take  advan- 
tage of  the  fact  that  the  use  of  evaluation  factors  to  evalu- 
ate technical  proposals,  when  required  by  the  nature  of  the 
requirement,  is  not  a  responsibility  determination.   Evaluation 
factors  have  been  used  on  requirements  where  highest  expertise 
or  best  scientific  approach  was  not  required.   This  is  indi- 
cated when  the  evaluation  scheme  contains  only  responsibility 
type  factors.   However,  GAO  has  condemned  the  practice  of 
rejecting  small  business  proposals  under  the  guise  of  relative 
assessment  in  order  to  avoid  compliance  with  statutory  require- 
ments to  submit  such  responsibility  determinations  to  SBA. 
I63;2] 
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V.   EVALUATION  OF  SERVICE  CONTRACTORS 

A.  PURPOSE 

''The  heart  of  the  evaluation  and  selection  process  is  the 
evaluation  factors  and  the  weights  assigned.   The  evaluation 
factors  should  consist  of  those  elements  which  the  Technical 
Evaluation  Panel  (TEP)  must  examine  in  each  proposal  in  order 
to  determine  an  offeror's:   [64:11] 

~   understanding  of  the  service  to  be  provided. 

~   technical,  business,  and  management  approach. 

-  potential  for  completing  the  job  as  specified  in 
RFP. 

-  probable  cost  based  upon  the  offeror's  approach. 

""   relative  qualifications  and  experience  including 
the  key  personnel  proposed. 

-  relative  competitive  status. 

-  coimnitment  and  assumption  of  risk. 

The  establishment  of  evaluation  factors  and  their  weights 
requires  the  exercise  of  judgment  on  a  case-by-case  basis 
I12;2-2J.   However,  it  is  the  purpose  of  this  chapter  to  pro- 
vide some  general  guidelines  on  the  selection  of  evaluation 
factors  and  then  some  more  specific  guidance  for  service 
contracts  in  particular. 

B.  GENERAL  GUIDELINES 

In  selecting  evaluation  factors  for  any  evaluation,  the 

evaluation  factors  should  meet  the  following  criteria.   The 

factors  should  be:   [65:50] 
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discriminating,  result  in  different  scores  for 
comji^eting  proposals  of  various  relative  merit. 

operational,  can  be  meaningfully  used  in  the  analysis. 

decomposable,  can  be  simplified  by  breaking  down  into 
unidimensional  parts, 

-  relevant  to  source  selection,  do  not  collect  "baseball 
statistics"  or  data  on  all  aspects  of  the  offeror. 

After  the  initial  list  of  factors  is  developed,  it  should 

be  tested  as  a  set  against  the  following  criteria:   [66:123] 

-  Independent  or  redundant,  double  counting  of  factors 
should  be  avoided, 

-  exhaustive  or  incomplete,  covers  all  the  important 
aspects  of  the  proposal. 

•-   compensating  or  unbalanced,  high  scores  in  one  factor 
offsets  low  scores  in  another. 


C.   SPECIFIC  GUIDELINES 

^ As  indicated  in  Chapter  IV,  there  are  no  restrictions  on 
the  kinds  of  evaluation  factors  which  may  be  used  as  long  as 
they  are  disclosed  in  the  RFP  and  relate  to  the  purpose  of 
the  procurement.   However,  a  review  of  major  weapons  systems 
, handbooks  and  guides,  and  interviews  with  NFAS  personnel  has 
shown  specific  areas  of  evaluation  common  to  service  contract- 
ing.  The  specific  factors  generally  fall  into  the  following 
jnajor  categories:   technical;  key  personnel;  management  and 
business;  experience  and  past  performance;  cost;  and  others. 
1.   Technical 

Often  contractors  will  be  required  to  submit  technical 
proposals  that  outline  their  scientific  or  engineering  approach 
to  a  problem  statement  set  forth  in  the  RFP,   The  SOW  is  very 
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broad  and  new,  innovative  ideas  are  encouraged.   The  technical 
factor  may  become  dominant  in  the  evaluation  scheme.   An 
example  of  how  the  major  technical  factor  may  be  decomposed  is 
as  follows:   [5:3-51] 

-  soundness  of  approach, 
understanding  of  the  requirement. 

-«   compliance  with  requirements  in  RFP. 

-  unique  ideas . 
2,   Key  Personnel 

The  SOW  may  also  be  very  detailed  and  specific  with  a 
fixed  approach.   An  example  would  be  training  services  with  a 
fixed  subject  matter  and  an  established  schedule  of  instruction 
periods.   The  source  selection  then  becomes  highly  dependent 
on  the  education,  experience,  and  quality  of  the  key  personnel 
proposed  by  each  offeror.   When  key  personnel  is  used  as  an 
evaluation  factor,  the  RFP  should  identify  those  functions  or 
positions  for  which  key  personnel  will  be  evaluated  and  require 
the  submission  of  resumes.   There  are  many  problems  related  to 
the  use  of  key  personnel  as  a  factor  and  the  submission  of 
resumes.   18] 

-  The  resumes  submitted  are  too  general  and  do 
not  address  the  specific  skills  or  experience 
required  for  the  immediate  procuremt, 

-  Th.e  key  personnel  proposed  are  not  currently 
employed  by  the  offeror  and  there  may  be 
uncertainty  as  to  their  availability. 

-  The  key  personnel  proposed  are  highly  experi- 
enced and  qualified  but  only  work  in  adminis- 
tration and  management  of  the  firm  and  rarely 
provide  direct  contract  effort  [67:21], 
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Certain  individuals  appear  on  multiple 
proposals. 

Key  personnel  may  depart  shortly  after  award 
of  the  contract  or  be  used  on  other  contracts. 

There  may  be  a  mismatch  between  the  qualifi- 
cations of  key  personnel  and  the  wages  reported 
in  the  cost  proposal. 

As  stated  in  Chapter  IV,  the  Government  cannot  require 
that  key  personnel  be  currently  employed  by  the  offeror  be- 
cause it  unduly  restricts  competition [44] .   Many  NFAS  actiti- 
ties  require  the  offerors  to  submit  letters  of  commitment 
from  key  personnel  with  their  proposal.   Several  activities 
have  developed  special  contract  clauses  that  require  that  key 
personnel  must  be  replaced  by  substantially  equal  personnel. 
Failure  to  do  so  then  becomes  a  grounds  for  Termination  for 
Default  I8J .    The  special  contract  clause  for  key  personnel 
used  by  NSC  Oakland  is  provided  as  figure  9 . 

Many  of  the  above  problems  common  in  the  use  of 
resumes  can  be  traced  to  inadequate  solicitations.   The  RFP 
must  clearly  identify  the  key  personnel  positions  and  provide 
explicit  direction  on  the  content  and  form  of  the  resumes. 
If  appropriate,  the  RFP  should  require  the  contractor  to  pro- 
vide personal  references  and  contact  points  in  order  for  the 
TEP  to  validate  an  individual ' s  performance  on  a  past  work 
effort.   Often  in  the  wording  of  resumes,  the  contractor  may 
overstate  or  conceal  the  relatively  minor  role  of  an  individual 
Also,  the  contractor  may  fail  to  state  that  the  customer  was 
dissatisfied  with,  the  performance  of  the  individual  [67:21]. 
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KEY  PERSONNEL  REQUIREMENTS  CLAUSE 

(a)   Certain  skilled  experienced  professional  and/or  technical 
personnel  are  essential  for  successful  contractor  accom- 
plishment of  the  work  to  be  performed  under  this  contract. 
These  are  defined  as  "Key  Personnel"  and  are  those  per- 
sons vrhose  resumes  were  submitted  for  evaluation  of  the 
proposal. 

The  contractor  agrees  that  such  personnel  shall  not  be 
removed  from  the  contract  work  or  replaced  without  com- 
pliance with  paragraphs  (b)  and  (c)  hereof. 

Cb).   If  one  or  more  of  the  key  personnel  for  whatever  reason 
becomes,  or  is  expected  to  become,  unavailable  for  work 
under  this  contract  for  a  continuous  period  exceeding  30 
work  days,  or  is  expected  to  devote  substantially  less 
effort  to  the  work  than  indicated  in  the  proposal  or 
initially  anticipated,  the  contractor  shall  immediately 
notify  the  Contracting  Officer  and  shall,  subject  to 
the  concurrence  of  the  Contracting  Officer  or  his  auth- 
orized representative,  promptly  replace  such  personnel 
with,  personnel  of  at  least  substantially  equal  ability 
and  qualifications. 

Ccl  All  requests  for  approval  of  substitutions  hereunder 

must  be  in  writing  and  provide  a  detailed ^explanation  of 
the  circumstances  necessitating  the  proposed  substitu- 
tions.  They  must  contain  a  complete  resume  for  the  pro- 
posed substitute,  and  any  other  information  requested 
by  tke  Contracting  Officer  or  needed  by  him  to  approve 
or  disapprove  the  proposed  substitution.   The  Contracting 
Officer  or  his  authorized  representative  will  evaluate 
such,  requests  and  promptly  notify  the  contractor  of  his 
approval  or  disapproval  thereof  in  writing. 

(d)   If  the  Contracting  Officer  determines  that  suitable  and 
timely  replacement  of  key  personnel  who  have  been 
reassigned,  terminated  or  have  otherwise  become  unavail- 
able for  the  contract  work  is  not  reasonably  forthcoming 
or  that  the  resultant  reduction  of  productive  effort 
would  be  so  substantial  as  to  impair  the  successful 
completion  of  the  contract  or  the  service  order,  the 
contract  may  be  terminated  by  the  Contracting  Officer 
for  default  or  for  the  convenience  of  the  Government, 
as  appropriate,  or,  at  the  discretion  of  the  Contracting 
Officer  if  he  finds  the  contractor  at  fault  for  the 
condition,  the  contract  price  or  fixed  fee  may  be 
equitably  adjusted  downward  to  compensate  the  Government 
for  any  resultant  delay,  loss  or  damage. 

USE:  In  RFP's  and  contracts  where  having  skilled  and  experi- 
enced technical  or  professional  personnel  constitutes  an 
important  factor  in  the  evaluation  of  technical  proposals 
for  award. 

Figure  9 
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If  the  TEP  is  not  satisfied  with,  the  resume  or  an  individual, 
the  oral  or  written  negotiations  required  to  be  held  with 
those  offerors  in  the  competitive  range  should  be  used  to 
gath.er  additional  data  [12:2-5].   If  the  TEP  simply  is  not 
satisfied  with  the  individual,  this  should  be  pointed  out  as 
a  deficiency  in  order  to  allow  the  contractor  the  opportunity 
to  propose  a  more  satisfactory  individual. 
3.   Management  and  Business 

An  example  of  a  further  breakdown  of  the  major  manage- 
ment and  business  factor  is  as  follows: 
management  and  business, 
management  plan. 

-  planning. 

-  organizing. 

-  controlling, 
corporate  resources. 

necessary  support  equipment. 

-  manpower  skill  mixes  and  number. 

-  necessary  support  facilities. 
Management  science  has  clearly  identified  planning, 

organizing,  and  controlling  as  the  three  basic  functions  of 
management  [68:92].   The  management  plan  describes  the 
offeror's  approach  for  efficiently  managing  the  work  and 
effectively  integrating  these  functions.   It  normally  includes: 
I12;2-4J 

'^   the  proposed  organization  including  internal 
operations  and  lines  of  authority. 
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external  relationships  and  interfaces  with 
the  Government. 

-  major  subcontractors. 

—  necessary  performance  schedules. 

The  management  plan  can  become  an  important  discrimi- 
nator in  contracting  for  services  for  the  operation  of 
Government  owned  facilities  and  systems.   Often  the  majority 
of  the  operational  personnel  remain  while  the  management 
groups  change  from  contract  to  contract.   Therefore,  the  pri- 
mary function  of  the  contractor  is  to  provide  management 
skills.   Ill] 

A  contractor  may  also  locate,  staff  and  equip  a 
facility  physically  close  to  a  major  Government  activity  in 
order  to  provide  support  services.   Examples  are  R&D  services, 
engineering  and  technical  services,  and  ADPE  services.   The 
discriminating  evaluation  factors  could  then  be  related  to 
superior  computer  capability,  superior  evaluation  and  test 
equipment,  or  superior  facility  features  [12:2-6].   In  this 
evaluation  area,  great  care  should  be  taken  to  separate  quali- 
fication factors  from  evaluation  factors.   For  example,  a  pro- 
curement may  require  that  a  contractor  provide  a  service  that 
necessitates  the  use  of  a  computer  compatible  with  Government 
computers.   The  computer  compatibility  requirement  should  be 
set  out  as  a  qualification  factor.   Evaluation  factors  for 
other  computer  compatibilities  may  or  may  not  be  used  in  the 
evaluation  plan.   If  the  qualification  factors  are  well  under- 
stood, contractors  lacking  the  required  personnel,  equipment, 
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or  facilities  are  not  likely  to  incur  the  expense  of  proposal 
preparation  [16:3-15], 

It  may  be  necessary  to  evaluate  personnel  as  a  corpor- 
ate resource  in  place  of  or  in  addition  to  key  personnel.   An 
e:^:ample  would  be  senior  engineers,  engineers,  draftsmen,  and 
technicians  required  to  provide  engineering  support  services. 
A  proper  skill  mijc  would  provide  adequate  services  without 
paying  for  over  qualified  personnel.   The  proper  numbers  of 
personnel  would  relate  to  the  schedule  being  offered. 

In  the  overall  area  of  corporate  resources,  if  an 
offeror  does  not  possess  adequate  resources  himself,  he  may  be 
able  to  demonstrate  the  ability  to  acquire  them  through  sub- 
contracts or  otherwise  [12:2-6] . 

4.   Experience  and  Past  Performance 

Past  performance  and  experience  are  often  used  inter- 
changeably but  there  is  a  difference  in  their  meaning.   Past 
performance  is  h.ow  well  an  offeror  did  on  earlier  work  and 
can  be  a  very  significant  indicator  of  how  well  he  can  be 
expected  to  perform  on  the  proposed  contract.   Many  organiza- 
tions exhibit  characteristics  that  persist  over  time.   Examples 
are  resiliency  in  the  face  of  trouble,  resourcefulness,  manage- 
ment determination,  skill  in  developing  key  personnel,  etc. 

Experience  is  the  accomplishment  of  work  by  an  offeror 
which  is  comparable  or  related  to  the  work  or  effort  required. 
Experience  includes  technical,  cost,  schedule,  and  management 
elements. 
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This  evaluation  factor  should  be  limited  to  the  over- 
all corporate  or  offeror  experience  and  past  performance  and 
not  the  past  performance  and  experience  of  individuals . 
Individuals  should  be  evaluated  under  key  personnel.   Informa- 
tion and  data  should  be  collected  in  addition  to  the  proposal 
submitted  by  the  offeror.   Admittedly,  the  assessments  to  be 
made  regarding  experience  and  particularly  past  performance 
may  be  very  difficult.   Often  the  data  will  not  be  clear 
because  of  the  difficulty  in  measuring  recorded  performance. 
In  the  event  a  substantially  unfavorable  response  is  received 
from  another  Government  activity,  the  offeror  should  be  given 
an  opportunity  durinu  discussions  to  submit  written  comments 
regarding  the  unfavorable  response.   [12:2-8] 

It  may  be  prudent  to  include  contractor  potential 
vrlth.  experience  and  past  performance.   New  corporations 
dissolve  and  reform  at  a  very  rapid  pace  and  it  is  not  uncommon 
to  be  faced  with  the  evaluation  of  a  highly  qualified  contrac- 
tor with  no  corporate  history.   Including  potential  provides 
the  TEP  with  considerable  flexibility.   [8] 

5.   Cost/Price 

Cost  or  price  is  a  required  factor  that  must  be  included 
in  the  evaluation  plan  and  relative  importance  revealed  in  the 
RFP.   However,  there  is  a  disagreement  as  to  whether  cost/price 
should  be  scored  in  the  same  manner  as  other  factors  or  whether 
cost/prica  should  be  evaluated  in  a  narrative  manner. 

From  a  theoretical  view,  there  is  no  difference  be- 
tween cost/price  and  any  other  factor  in  the  evaluation  plan. 
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It  must  be  converted  to  the  same  utility  scale  in  order  to 
evaluate  the  trade-offs  between  factors  [24] .   Most  NFAS 
activities  score  cost/price  using  a  variety  of  scoring 
methods  [8]  . 

The  GAO  prefers  that  cost/price  not  be  scored.   They 
prefer  that  after  a  complete  technical  evaluation,  the  con- 
tracting officer  or  source  selection  official  should  prepare 
a  narrative  justification  for  the  expenditure  of  funds  over 
and  above  that  proposed  by  the  lowest  qualified  offeror  in 
the  competitive  range.   The  rationale  behind  this  position  is 
that  all  factors  other  than  cost/price  are  judgmental  and 
therefore  subjective.   On  the  other  hand,  cost/price  is  a 
totally  objective  value  and  scoring  distorts  an  already  clear 
value,  llost  major  weapons  systems  activities  and  NASA  do  not 
score  cost.   Because  this  issue  is  not  addressed  in  statute, 
GAO  accepts  both  methods.   [69] 

By  whatever  method  cost/price  is  to  be  evaluated,  a 
clear  determination  must  be  made  as  to  what  elements  are  to 
be  included  in  the  cost/price  evaluation.   Certain  elements 
are  required  by  regulations  and  must  be  considered  in  the 
evaluation  even  if  they  are  not  listed  in  the  RFP.   Examples 
are  fright  costs,  transportation  cost  and  Government  Furnished 
-Material  CGFM)  .   149] 

Options  may  be  included  in  a  service  contract  if  there 
is  an  anticipated  need  beyond  the  initial  contract  period. 
This  isL  in  recognition  of  the  Government's  need  for  continuity 
of  operations  and  to  preclude  cost  associated  with  disrupted 
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support.   Normally  evaluations  will  be  made  exclusive  of  the 
option  period.   DAR  7-2003.11  provide   special  clauses  for  the 
evaluation  of  options  and  to  discourage  the  submission  of 
unbalanced  proposals.   An  unbalanced  proposal  is  one  which  is 
based  on  prices  significantly  less  than  cost  for  one  period 
and  prices  significantly  more  than  cost  for  another  period. 
An  unbalanced  proposal  may  be  rejected  as  non-responsive. 

The  option  periods  may  be  considered  in  fixed  price 
type  contracts  if  realistic  competition  is  impracticable  after 
the  initial  contract  is  awarded.   This  may  be  a  result  of  sub- 
stantial startup  or  phase-in  costs  or  superior  technical 
ability  resulting  from  performance  of  the  initial  contract. 
Options  may  also  be  considered  if  there  is  a  known  requirement 
beyond  the  initial  period  and  funding  is  currently  unavailable. 
A  reasonable  certainty  must  exist  that  funds  will  be  available 
to  exercise  the  option.   [70:1-1500] 

Start-up  or  phase-in  cost  to  the  Government  may  be 
assigned  to  all  proposals  other  than  the  incumbent  contractor's 
proposal  if  specifically  set  forth  in  the  RFP  [48] .   This 
naturally  favors  the  incumbent  and  the  procurement  activity 
should  expect  to  defend  the  amount,  relevance  and  need  of  the 
costs  assigned. 

D.   TESTING  THE  EVALUATION  FACTORS 

The  evaluation  factors  may  be  tested  for  appropriateness 
by  using  different  perspectives.   From  the  potential  contrac- 
tor's perspective,  the  evaluation  factors  must  make  sense  in 
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terms  of  the  SOW  requirements.   From  the  TEP  perspective,  there 
should  be  a  match  between  the  evaluation  factors  and  the  skills 
and  expertise  of  the  TEP.   The  TEP  must  be  able  to  develop 
documentation  to  justify  the  differences  between  acceptable 
offerors  (lowest  acceptable  to  highest  acceptable) .   From  the 
contractijig  officer's  perspective  the  evaluation  factors  must 
not  violate  the  rules  summarized  in  Chapter  IV.   The  evaluation 
factors  must  also  be  viewed  from  the  perspective  of  future 
unsuccessful  offerors.   Although  not  violating  any  particular 
regulation  or  rule,  the  evaluation  factors  may  unnecessarily 
invite  protest. 

E.   THE  FINAL  EVALUATION  PLAN 

The  RP9  is  only  required  to  reveal  the  major  evaluation 
factors  and  their  relative  importance.   However,  a  final  and 
complete  evaluation  plan  should  be  developed  early  to  avoid 
accusations  that  the  evaluation  plan  favors  certain  offers. 
This  may  be  prior  to  the  release  of  the  RFP,  but  certainly 
prior  to  the  opening  of  any  proposals.   The  evaluation  plan 
should  contain  the  weights,  scoring/rating  system  and  the 
basis:  for  recommendation  for  award.   Some  complex  normalization 
procedures  have  been  used  in  an  attempt  to  increase  accuracy. 
For  example,  the  measure  of  standard  deviations  from  the 
average  technical  score  as  a  factor  to  apply  to  raw  scores. 
Any  reasonable  system  is  acceptable  as  long  as  it  does  not  give 
unreasonable  results.   However,  a  system  such  as  MAUM  is  sup- 
ported by  sound  decision  theory  and  extensive  field  research. 

[2:7] 
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The  issue  of  scoring  cost  is  very  sensitive  and  should  be 
settled  by  making  it  a  part  of  the  evaluation  plan.   In  the 
NFAS  activities  interviewed,  the  cost  proposal  is  removed  from 
the  offeror's  proposal  prior  to  the  initial  technical  evalua- 
tion by  the  TEP.   The  cost  proposal  is  evaluated  by  the  con- 
tracting personnel  and  the  cost  proposal  is  later  provided  to 
the  TEP  to  check  compatibility  and  reasonableness  with  the 
technical  proposal.   The  cost  proposal  is  scored  by  the  con- 
tracting personnel  often  working  closely  with  the  TEP  [81] . 
Local  activities  may  develop  special  methods  for  assigning 
points  to  cost.   Two  such  methods  are  the  Twice-Low  Method  and 
the  Greatest  Value  Method. 

Twice-Low  Method;   NSC,  Oakland  has  developed  the  Twice- 
Low  Method  to  assign  points  for  cost  factors.   The  method  is 
best  explained  by  an  example.   If  a  maximum  of  20  points  was 
possible,  the  lowest  acceptable  offer  would  receive  the  full 
2Q  points.   The  cost  proposal  of  the  lowest  offer  is  then 
doubled  and  any  offeror  with  a  cost  proposal  greater  than  that 
amount  would  receive  a  score  of  zero.   Any  cost  proposal  within 
this  range  is  scored  on  a  straight  linear  scale.   Therefore,  a 
cost  proposal  halfway  in  between  would  receive  a  score  of  10. 
I71J  . 

Greatest  Value  Method:   NRCO,  Long  Beach  has  developed 
the  Greatest  Value  Method.   In  this  method  the  cost  proposal 
of  the  lowest  offeror  is  divided  by  the  cost  proposal  of  the 
of;geror  being  evaluated  and  then  multiplied  by  the  maximum 
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points  possible.   This  also  results  in  the  lowest  offeror 
receiving  the  maximum  score  possible.   Therefore,  if  low 
acceptable  offeror  A  proposed  $150,000  and  next  offeror  B 
proposed  $200,000,  of  a  possible  30  points,  offeror  A  would 
receive  30  and  offeror  B  would  receive  22.5. 

Both  activities  recognize  that  these  methods  are  only 
tools  and  that  each  point  assignment  must  be  supported  by 
written  narrative  judgment.   Variations  from  using  these 
mathematical  formulas  are  allowed  but  must  also  be  properly 
supported  in  writing. 

The  TEP  and  contracting  personnel  should  clearly 
establish,  the  basis  of  award  recommendation.   Normally,  the 
offeror  with,  the  highest  combined  technical  and  cost  score  is 
recommended  for  award.   However,  award  may  be  on  a  different 
basis.   For  example,  award  may  also  be  made  to  the  acceptable 
offeror  in  the  competitive  range  with  the  lowest  cost/price 
proposal.   The  basis  for  such,  an  award  must  be  set  forth  in 
the  RFP.   159] 

F.   PROPOSAL  PREPARATION  GUIDANCE 

After  careful  preparation  of  the  SOW  and  SSP,  the  only 
major  component  remaining  prior  to  release  of  the  RFP  is  the 
Proposal  Preparation  Guidance  (PPG) .   Again,  it  is  important 
that  the  PPG  be  consistent  with  the  SOW  and  evaluation  factors 
in  the  RFP.   Explicit  instructions  in  format  as  well  as 
implicit  guidance  contained  in  the  organization  and  layout  of 
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the  RFP  indicate  to  the  offeror  what  the  Government  would  like 
to  see  in  the  proposal. 

Proposals  are  expensive  and  time  consuming  for  the  offeror 
to  prepare  and  for  the  Government  to  evaluate.   While  proposals 
are  expected  to  be  comprehensive,  excess  volume,  detail  and 
farochuremanship  do  not  serve  either  party.   Misunderstandings 
and  needless  expenses  are  avoided  if  the  RFP  clearly  estab- 
lishes what  is  important  and  how  to  prepare  proposals.   To 
control  size,  the  Government  must  exercise  restraint  in  the 
listing  of  requirements  and  in  the  detail  demanded.   The  esti- 
mated value  of  the  contract  should  be  used  to  strike  a  balance 
between  what  the  Government  must  have  and  what  would  be  nice  to 
have. 

The  RFP  should  direct  the  offeror  how  to  organize  and 
arrange  the  proposals.   This  will  not  only  facilitate  the  evalu- 
ation tasks  of  the  TEP  but  will  also  facilitate  proposal  prepa- 
ration.  Great  care  should  be  taken  in  drafting  these 
instructions  in  order  that  desirable  innovations  are  not  pre- 
cluded 116:3-16]..   The  solicitation  should  include  only 
necessary  formats  and  concepts  rather  than  an  elaborate  instruc- 
tion for  preparing  proposals.   The  offerors  should  be  directed 
to  submit  a  sufficient  number  of  copies  of  their  proposal  in 
order  that  each  TEP  member  shall  have  his  own  copy. 

The  RFP  should  have  a  specific  list  all  in  one  place  of 
the  data  needed  in  the  proposal.   The  list  should  be  compre- 
hensive and  not  depend  on  the  offeror  to  fill  in  the  blanks. 
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The  instructions  should  motivate  the  contractor  to  submit  a 
"First  and  Final"  proposal  and  not  to  depend  on  negotiations 
to  correct  any  deficiencies  [7] . 

G.   PREPROPOSAL  CONFERENCE 

Since  preproposal  conferences  are  expensive  and  time  con- 
suming to  host  and  attendees,  they  should  be  used  sparingly. 
A  preproposal  conference  may  be  necessary  in  order  to  provide 
information  that  cannot  be  provided  in  the  RFP  or  to  clarify 
nebulous  points.   Examples  are  tours  of  Government  facilities 
to  be  operated  or  formal  presentations  to  further  explain 
complex  requirements.   [5:4-20] 

Wh.en  the  decision  has  been  made  to  hold  a  preproposal 
conference,  the  procurement  personnel  make  the  necessary 
arrangements.   Adequate  notice  shall  be  given  to  prospective 
offerors  in  order  that  tKey  may  submit  questions  and  make 
arrangements,  to  attend.   The  exact  nature  and  scope  of  che 
conference  should  be  explicitly  stated. 

The  conference  itself  should  be  conducted  by  the  contract- 
ing officer  or  a  contract  specialist.   The  TEP  will  assist  and 
should  be  briefed  on  what  can  and  cannot  be  discussed.   Care 
should  be  taken  to  place  emphasis  consistent  with  the  evalua- 
tion factors  and  SOW.   All  offerors  should  be  provided  identi- 
cal information  and  advised  that  unless  the  RFP  is  amended  in 
writing,  it  remains  unchanged.   A  complete  record  shall  be 
made  of  the  preproposal  conference  and  made  a  part  of  the  con- 
tract file, 
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VI.   CONDUCTING  THE  EVALUATION 

A.  PURPOSE 

The  purpose  of  this  chapter  is  to  list  and  describe  the 
essential  steps  of  the  evaluation  process.   It  is  at  this 
point  that  the  extra  time  and  effort  invested  in  planning  pays 
off.   The  primary  function  of  the  evaluation  process  is  to  pro- 
vide a  sound  recommendation  to  the  contracting  officer  in 
order  that  he  may  make  an  informed  and  objective  source  selec- 
tion.  This  may  consist  of  a  single  evaluation,  but  more  often 
additional  data  gathering,  negotiations,  revised  proposals 
and  a  second  evaluation  are  required.   This  makes  the  process 
a  combination  of  factfinding,  reporting  and  application  of 
professional  judgment.   [16:3-24] 

B.  TEP  FAMILIARIZATION 

Just  prior  to  the  receipt  of  proposals  from  the  contract- 
ing officer  each  Technical  Evaluation  Panel  (TEP)  member 
should  review  the  Request  for  Proposal  (RFP)  to  clearly 
establish  how  th.e  Statement  of  Work  CSOW)  was  presented  and 
exactly  how  the  evaluation  criteria  relate  to  this  require- 
ment.  Several  months  may  have  passed  and  solicitation  amend- 
ments may  have  been  issued.   Each  evaluator  must  understand 
that  the  evaluation  is  to  be  accomplished  in  accordance  with 
the  RFP  as  written.   What  each  evaluator  thinks  it  should  say 
cannot  be  allowed  to  enter  the  evaluation  process. 
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A  meeting  of  the  TEP  should  be  held  to  review  and  update 
the  SSP.   The  TEP  chief  should  chair  the  meeting.   The  primary 
purpose  of  the  meeting  is  to  cover  all  the  administrative 
details  of  conducting  the  evaluation  and  to  establish  a 
schedule  for  evaluation  completion  [5:4-22].    Ideally,  TEP 
members  should  be  divested  of  all  other  duties  and  provided 
a  separate  working  area  to  conduct  the  evaluations  [16:2-7]. 
However,  most  NFAS  activity  customers  are  unable  to  do  this 
and  each  TEP  member  must  balance  TEP  duties  with  regular 
office  duties.   These  problems  must  be  confronted  and  solutions 
negotiated  because  the  longer  the  evaluation  drags  out  the 
greater  the  chance  that  either  the  cost  proposals  will  expire 
and  therefore  need  revision  or  the  confidentiality  of  a  pro- 
posal will  be  violated  [11] . 

C.   RECEIPT  AND  BREAKOUT  OF  PROPOSALS 

DAR  does  not  have  a  detailed  procedure  for  the  receipt, 
storage,  and  opening  of  proposals  received  in  a  negotiated 
procurement.   However,  the  NFAS  activities  interviewed  have 
well-developed  procedures.   All  proposals,  revised  proposals, 
letters,  telegrams  and  other  correspondence  are  received  by 
the  contracting  office.   The  original  documents  are  made  a 
permanent  part  of  the  contract  file  and  only  copies  are  dis- 
tributed to  the  TEP.   The  cost  proposals  are  usually  separated 
from  th.e  technical  proposal  and  provided  to  the  TEP  only  after 
the  initial  technical  evaluation  is  completed.   A  few  activi- 
ties have  gone  so  far  as  to  mask  the  identity  of  each  offeror 
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in  order  to  increase  objectivity.   The  proposals  are  forwarded 
to  the  terhnical  activity  by  U.  S.  mail  with  a  cover  letter 
that  includes  the  appropriate  cautions  [11] . 

Technical  activities  have  a  less  structured  proposal  con- 
trol procedure.   Classified  proposals  are  controlled  by  the 
same  security  procedure  as  all  classified  documents.   Due  to 
the  collateral  nature  of  the  TEP  position,  unclassified  evalu- 
ations are  often  conducted  at  home  during  the  evenincr  or  on 
weekends.   Security  and  efficiency  can  be  improved  if  each 
evaluator  has  a  complete  set  of  proposals.   Each  evaluator  can 
then  be  individually  accountable  for  a  particular  set  of  pro- 
posals.  If  for  some  reason  proposals  must  be  routed,  delivery 
is  by  hand  and  not  by  guardmail.   [11] 

D.   INITIAL  EVALUATION 

1.  Qualification  Factors 

The  initial  evaluation  start  with  a  review  of  each 
proposal  using  the  qualification  factors.   This  is  a  very 
mechanical  process  with  a  strict  set  of  rules.   The  objective 
facts  make  a  contractor  either  qualified  or  not  qualified  and 
professional  judgment  is  seldom  involved.   Proposals  that  do 
not  meet  the  qualification  factors  must  be  removed  from  compe- 
tition or  the  procurement  activity  is  open  to  protest  from 
competitors . 

2.  Grossly  Deficient  Proposals 

During  this  initial  review,  any  proposal  that  is  so  grossly 
deficient  that  it  does  not  provide  a  reasonable  basis  on  which 
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to  conduct  an  initial  evaluation  without  a  total  rewrite  may 
be  rejected  as  technically  unacceptable  [55]  .   Surprisingly, 
incumbent  offerors  may  assume  that  the  Government  is  familiar 
with  their  capabilities  and  decide  not  to  incur  the  expense 
of  a  detailed  proposal.   Evaluation  panels  have  rejected  pro- 
posals from  incumbents  as  grossly  deficient  and  eliminated 
them  from  further  consideration.   Grossly  deficient  proposals 
are  from  contractors  that  do  not  have  a  basic  understanding 
of  Government  contracting  in  general  or  of  the  anticipated 
work  effort  in  particular.   It  is  not  the  function  of  the  TEP 
to  provide  basic  lessons  in  the  proper  preparation  of  proposals 
I.15J  .   A  note  of  caution  must  be  taken  to  ensure  that  the 
problem  is  with  the  offeror  and  not  the  result  of  a  grossly 
deficient  RFP  [9] . 

E.   IN-DEPTH  EVALUATION 

1.   Considerations  During  Evaluation 

There  is  a  lot  of  gamesmanship  and  brochuremanship 
used  in  the  preparation  of  proposals  by  offerors.   Some  points 
to  consider  when  performing  a  technical  evaluation  are: 
I6;rv-7J 

-  avoid  "reading  into"  or  "reading  out  of"  thereby 
overly  interpreting  what  is  written. 

—  avoid  the  influence  of  first  impressions. 

~   look  for  ambiguities  and  inconsistencies. 

recognize  the  use  of  "catch  phrases"  and  "buzz 
words. " 
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recognize  the  difference  between  technical 
substance  and  glossy  presentation  format. 

-  recognize  the  existence  of  flattery  by  an 
offeror. 

Some  of  the  games  that  offerors  may  play  are: 

[12:127] 

-  on  a  cost  type  contract  an  offeror  may  overstate 
his  future  business  base  in  order  to  substantiate 
a  relatively  low  overhead  rate. 

-  on  a  cost  type  contract  the  offeror  may  optimisti- 
cally underestimate  the  direct  labor  hours  for 

a  function  such  as  engineering  design  in  order 
to  propose  the  lowest  estimate  cost. 

-^   an  offeror  may  hold  a  lower  Best  and  Final  figure 
for  use  at  the  last  moment  in  an  effort  to  show 
a  competitive  attitude  or  in  hopes  of  obtaining 
last  minute  knowledge  of  what  the  competition 
is  doing. 

-  an  offeror  may  hire  a  competitor's  employee  in 

an  effort  to  validate  or  enhance  his  knowledge  of 
the  competition. 

-  an  offeror  may  attempt  to  "buy-in"  with  hopes  of 
making  profits  on  changes. 

2,   Errors,  Omissions  and  Deficiencies 

While  conducting  the  evaluation  the  TEP  may  encounter 
errors,  omissions  and  deficiencies,  some  of  whicn  are  super- 
ficial and  easily  corrected  while  others  are  more  serious. 
Errors  are  a  mistake  in  calculation  or  measurement  or  a  minor 
misconception.   An  omission  is  simply  a  failure  to  respond  to 
a  particular  requirement  in  the  RFP. 

A  deficiency  may  be  any  part  of  an  offeror's  proposal 
which.;   116:3-24] 
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fails  to  meet  the  minimum  requirements  of  a 
standard  set  forth  in  the  RFP. 

proposes  an  approach  that  has  unacceptable 
risk. 

describes  an  approach  taken  by  the  offeror 
which  yields  undesirable  performance. 

3.   The  Evaluation 

As  stated  in  Chapter  III,  after  all  the  efforts  to 
develop  a  comprehensive  and  effective  evaluation  structure, 
the  individual  evaluator  is  still  faced  with  the  problem  of 
exactly  how  to  evaluate  the  proposals.   The  problem  centers 
on  assigning  scores  in  such  a  manner  as  to  maintain  consist- 
ency for  each  evaluator  and  comparability  between  the  evalua- 
tors.   Evaluators  must  not  make  the  mistake  of  rating  the 
proposals  in  relation  to  each  other.   Individual  evaluators 
tend  to  choose  a  single  proposal  as  a  base  and  rate  other 
proposals  in  relation  to  the  base  proposal.   Other  evaluators 
may  use  a  mixture  of  proposals  as  a  base  or  not  have  a  base. 
Bacause  the  individual  evaluators  are  not  evaluating  to  the 
same  base,  their  findings  are  less  consistent  and  comparable. 

A  well  written  RFP  should  have  clearly  established 
standards  and  serve  as  a  single  basis  against  which  to  compare 
proposals.   The:  oversimplified  procedure  is  to  take  each  eval- 
uation factor  and  compare  the  proposal  against  the  standard  in 
the  RFP.   Using  his  judgment  and  expertise,  the  variance  is 
determined  by  the  evaluator,  described  in  narrative  form,  and 
represented  in  numerical  form.   [14:27] 

Care  must  also  be  taken  to  ensure  that  the  evaluation 

does  not  center  just  on  errors,  omissions  and  deficiencies. 
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Positive  factors  that  are  above  the  standards  can  offset 
other  negative  factors.   Therefore,  it  is  necessary  to 
consider  not  only  what  is  positive  or  negative,  but  also  the 
degree  to  which  it  is  good  or  bad.   In  the  case  of  negative 
factors,  the  difficulty  of  correcting  the  factor  must  be 
fully  assessed.   The  impact  of  a  negative  factor  may  be  so 
great  as  to  make  the  entire  proposal  unacceptable  or  so  minor 
as  to  be  insignificant.   In  the  case  of  a  positive  element, 
the  evaluator  must  determine  not  only  the  excess  above  the 
standard,  but  also  whether  the  excess  represents  a  useful 
value  to  the  Government.   [14:29] 

4.   Requests  for  Additional  Information 

The  TEP  has  a  natural  desire  to  start  the  negotiations 
before  they  get  done  with  the  initial  evaluation  [11] .   TEP 
members  are  frustrated  by  what  appear  to  be  simple  omissions 
and  errors  in  the  proposals.   Regulations  permit  the  TEP  to 
obtain  clarifications  from  the  offeror  during  the  evaluation 
process.   The  evaluators  prefer  to  request  these  clarifica- 
tions as  they  arise  claiming  the  evaluation  cannot  proceed 
until  they  are  clarified.   This  can  result  in  a  continuous 
stream  of  questions  from  the  TEP  and  answers  from  the  offerors 
flowing  through  the  contracting  office.   The  combined  result 
is  the  conversion  of  clarifications  into  negotiations  and 
extensive  delays  in  completion  of  the  initial  evaluation  [7] . 
The  TEP  must  at  least  exercise  restraint  and  NASA  has 
gone  so  far  as  to  forbid  any  questions  until  negotiations  are 
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formally  initiated.   NASA  evaluators  are  directed  to  complete 
evaluations  based  on  the  most  reasonable  assumptions  they  can 
make  regarding  the  proposal  areas  involved.   Their  assumptions 
and  th-e  additional  information  are  the  subject  of  oral/written 
discussions  with  those  firms  determined  to  be  in  the  competi- 
tive range.   This  has  the  added  advantage  of  avoiding  unneces- 
sary communication  with  offerors  outside  the  competitive 
range.   120:3-6] 

5,   Cost/Price  Considerations 
a.   Cost/Price  Analysis 

Regardless  of  the  type  of  contract,  the  relative 
value  of  cost/price,  or  whether  cost/price  is  scored  or  not 
scored,  a  complete  analysis  of  the  cost/price  proposal  is 
required.   It  is  at  this  point  that  the  differences  in 
resources  available  to  the  major  weapons  systems  activities 
and  tKe  NFAS  activities  become  significant.   The  major  weapons 
systems  activities  often  have  professional  cost/price  analysts 
that  provide  support  to  the  contracting  officer  and  TEP. 
Only  a  few  cost/price  analysts  are  currently  assigned  to  NFAS 
activities.   At  the  NFAS  activities  interviewed,  the  cost/price 
analyst  function  was  performed  by  contract  specialists  with 
primary  support  from  the  Defense  Contract  Audit  Agency  DCAA) 

A  DCAA  audit  is  very  time  consuming  with  sixty  day 
deliveries  being  common  at  the  NFAS  activities  interviewed. 
The  NFAS  activities  recognize  that  much  of  this  delay  is 
caused  by  excessive  backlog  at  DCAA  and  therefore  take  measures 
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to  limit  the  number  of  DCAA  audit  requests.   The  primary 
method  is  to  delay  the  request  for  DCAA  audit  until  after 
the  TEP  has  completed  a  preliminary  evaluation  using  qualifi- 
cation factors  and  searching  for  grossly  deficient  proposals. 
This  eliminates  the  need  for  DCAA  to  conduct  an  unnecessary 
audit  on  an  offeror  that  does  not  have  a  chance  for  award. 
Another  method,  used  primarily  on  smaller  service  contracts, 
is  to  limit  the  DCAA  audit  to  a  rate  verification  conducted 
through,  the  resident  DCAA  Liaison  Office.   A  rate  verification 
can  be  completed  in  only  days  and  is  quite  sufficient  if  the 
offeror  has  a  substantial  volume  of  Government  business , 
iQate.rial  is  not  involved  and  rates  have  not  changed.   However, 
if  a  significant  problem  in  rates  is  uncovered,  a  complete 
audit  is  necessary  18] .   The  last  method  available  to  reduce 
the  number  of  DCAA  audit  requests  is  to  wait  until  after  a 
complete  technical  evaluation.   This  will  eliminate  those 
offerors  that  are  not  technical  acceptable.   The  goal  of  the 
NFAS  activities  is  to  hold  the  number  of  DCAA  audit  requests 
per  evaluation  to  no  more  than  three,  if  possible  [4] . 

Another  major  aid  in  the  cost/price  analysis  is 
the  independent  cost  estimate  CICE)  developed  as  part  of  the 
SSP.   Preparation  of  an  ICE  is  discussed  in  Chapter  II.   A 
well  prepared  ICE  adds  another  dimension  to  the  cost/price 
analysis  and  can  be  a  critical  aid  if  action  must  be  taken 
without  the  benefit  of  a  DCAA  audit.   Unfortunately,  the 
ability  or  resources  of  most  NFAS  activities  and  their 
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customers  in  ICE  preparation  is  limited.   Furthermore,  an  ICE 
may  be  inaccurate  due  to  the  performance  uncertainties  inherent 
particularly  in  cost  type  contracts.   As  a  result,  GAO  has 
cautioned  against  undue  reliance  on  the  ICE.   [47] 

Another  cost/price  analysis  method  is  to  compare 
the  cost  proposals  of  all  offerors.   For  fixed  price  require- 
ments that  do  not  have  evaluation  factors,  this  may  be  the 
entire  basis  for  the  contracting  officer's  determination  that 
the  price  is  fair  and  reasonable.   However,  cost/price  analysis 
has  a  different  objective  when  used  in  conjunction  with  require- 
ments that  have  evaluation  factors.   That  objective  is  source 
selection  and  a  fair  and  reasonable  price  determination.   The 
GAO  has  found  that  when  the  cost/price  analysis  consists  of 
merely  comparing  the  estimated  cost  proposals  submitted  by 
offerors,  rational  support  for  the  source  selection  does  not 
exist  due  to  inadequate  analysis  and  assessment.   [46] 

Cost/price  analysis,  as  required  by  DAR  3.807.2, 
may  be  accomplished  in  various  ways  including  DCAA  audit, 
comparison  against  an  ICE,  and  comparison  of  proposals  sub- 
mitted.  GAO  encourages  the  use  of  all  information  available 
at  the  time  of  evaluation.   [54] 

b.   Cost  Realism/Probable  Cost 

It  is  fundamental  in  the  award  of  a  cost  type 
contract  that  proposed  cost  be  analyzed  in  terms  of  cost 
realism  14  5]  ^   At  the  NFAS  activities  interviewed,  cost 
realism  was  considered  after  completion  of  the  technical  evalu- 
ation by  the  TEP  and  completion  of  the  cost  analysis  by  the 
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contracting  personnel.   By  comparing  the  technical  proposal 

with  the  cost  proposal,  the  TEP  and  contracting  personnel 

determine:    [14:52] 

-   that  the  offeror's  cost  estimate  is  compatible 
with  the  technical  and  management  effort 
proposed. 

that  all  costs  have  been  included  and  that 
estimated  costs  are  relatively  valid. 

that  the  costs  are  in  consonance  with  the 
work  statements  in  the  RFP. 

Significant  inconsistencies  between  the  technical 
proposal  and  the  cost  proposal  can  be  considered  in  two  ways. 
The  inconsistencies  may  raise  a  fundamental  issue  concerning 
thjB  offeror's  understanding  of  the  nature  and  scope  of  the 
work,  required.   This  may  result  in  appropriate  areas  of  the 
technical  evaluation  being  re-scored.   The  evaluators  may  be 
able  to  determine  a  more  probable  cost  than  that  proposed  by 
the  offeror  I6;IV-6J.    It  is  then  appropriate  to  use  the 
probable  cost  to  score  or  evaluate  the  cost  proposal  [58]  . 
Any  significant  inconsistency  will  be  the  subject  of  negotia- 
tions: and  th.e  burden  of  proof  as  to  the  credibility  of  the 
proposed  cost  rests  with  the  offerors. 

Cost  realism  can  also  be  an  important  factor  in 
fixed  price  contracting  and  a  defense  against  the  "buy-in." 
There  is  nothing  illegal  about  a  "buy-in"  and  it  is  not  an 
acceptable  basis  for  rejecting  a  proposal  [9].   However,  lack 
of  cost  realism  can  affect  the  technical  evaluation  in  much 
the  same  was  as  in  cost  type  contracting.   Also,  if  cost  is 


85 


being  separately  scored,  a  cost  proposal  can  be  scored  lower 
for  lack  of  cost  realism.   A  few  individuals  interviewed 
cautioned  against  being  too  heavy  handed  and  attempting  to 
declare  an  unusually  low  offeror  as  unacceptable  based  solely 
on  lack  of  cost  realism.   Much  less  justification  is  required 
for  reducing  an  offeror's  score  than  for  removing  the  offeror 
from  further  consideration.   In  addition,  a  reduced  score  is 
much  less  likely  to  provoke  a  protest.   [8] 

F.   RESULTS  OF  INITIAL  EVALUATION 
1.   Award  on  Initial  Proposals 

Part  of  the  boilerplate  of  any  contract  is  a  statement 
that  award  may  be  made  on  the  basis  of  initial  proposals. 
There  are  two  basic  situations  that  may  occur  where  award  on 
tha  basis  of  initial  proposals  may  be  appropriate.   The  urgency 
of  the  delivery  requirements  may  not  permit  negotiations  or 
negotiations  simply  are  not  necessary.   The  Government  must 
demonstrate  that  adequate  competition  does  exist  and  that 
acceptance  of  the  most  favorable  initial  proposal,  without 
further  discussion,  would  result  in  a  fair  and  reasonable  cost/ 
price  to  the  Government.   This  is  proper  only  if  the  proposal 
is  accepted  exactly  as  submitted  without  any  negotiations 
whatsoever . 

The  decision  to  award  on  the  basis  of  an  initial  pro- 
posal is  within  the  discretion  of  the  Government.   Although 
the  above  conditions  may  be  met,  the  Government  may  choose  to 
negotiate  for  a  variety  of  reasons.   The  regulations  do  not 
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require  an  award  without  discussion,  they  only  permit  such  an 
award  [40:4].    Individuals  interviewed  stated  that  award  on 
the  basis  of  initial  proposals  is  not  common.   However,  they 
considered  it  important  that  the  offeror  fully  realize  that 
the  possibility  of  award  on  the  basis  of  initial  proposal  does 
exist.   This  forces  the  offeror  to  put  more  effort  into  the 
initial  submission  and  not  depend  on  the  negotiation  process 
to  provide  clues  as  to  the  Government's  true  desires.   Also, 
it  forces  the  offeror  to  submit  a  more  realistic  cost/price 
proposal  and  not  hold  back  for  the  Best  and  Final  (B&F) .   The 
NFAS  activities  interviewed  routinely  conduct  negotiations  and 
request  a  B&F  and  do  not  actively  consider  award  on  the  basis 
of  initial  proposals  [11] . 

2.   First  Technical  Evaluation  Report  (TER) 

The  primary  objective  of  the  TER  is  to  clearly  and 
objectively  state  the  results  of  the  proposal  evaluation  by 
the  TEP .   This  documentation  is  the  only  official  record 
showing  tha  logic  and  rationale  used  by  the  TEP  members  when 
reaching  their  conclusion.   A  brief  discussion  of  significant 
weaknesses  and  strengths  is  to  be  provided  to  support  the 
score  assigned  under  each  evaluation  factor.   It  is  important 
that  a  substantive  TER  be  created  in  order  to:   [5:4-25] 

-  provide  the  contracting  officer  with  the  necessary 
information  in  order  to  notify  unsuccessful 
offerors. 

-  serve  to  refresh  the  memories  of  TEP  members 
during  later  debrief ings  with  rejected  offerors. 
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to  become  the  official  documentation  should 
the  award  be  questions. 

support  the  competitive  range  determination. 

aid  in  the  development  of  negotiation  objectives. 

Naturally,  if  the  TEP  is  recommending  an  award  based 
on  initial  proposals,  there  is  no  need  for  the  development 
of  negotiation  objectives  or  a  competitive  range  determination. 
However,  it  would  be  prudent  to  document  the  reasons  why 
further  negotiations  are  not  required  or  in  the  best  interests 
of  the  Government.   Contractors  often  challenge  either  form- 
ally or  informally,  an  award  based  on  initial  proposals  [11]. 

While  there  is  no  standard  format  for  a  TER,  certain 
generalizations  can  be  made.   [5:4-26] 

—  Findings  must  be  supported  by  rational  judgment 
or  technical  data. 

—  Supporting  data,  findings,  technical  analysis 
and  judgment  statements  are  to  be  presented  so 
people  with  nontechnical  backgrounds  can  under- 
stand as  much  as  possible. 

—  ThB  selection  of  words  should  be  consistent 
with,  the  findinds  of  the  TEP.   An  outstanding 
offer  should  be  described  in  superlative  terms 
rather  than  mediocre  terms. 

~   The  TER  is  a  summary  and  great  bulk  is  not 

required  or  desired.   All  supporting  data  is  to 
be  retained  as  part  of  the  contract  file. 

~   A  primary  function  of  the  TER  is  to  categorize 
each  proposal  as  either  acceptable,  unacceptable 
but  capable  of  being  made  acceptable  without 
major  revision,  or  unacceptable. 

This  is  in  preparation  for  the  competitive  range 

determination  to  be  made  by  the  contracting  officer.   Offerors 

with,  unacceptable  proposals  are  notified  by  the  contracting 

88 


officer  and  are  no  longer  considered.   Each  of  the  remaining 
offerors  are  then  ranked  and  it  is  important  that  the  relative 
rank  be  clearly  stated.   Differences  among  acceptable  proposals 
are  critical  factors  in  the  subsequent  decisions  related  to 
selection  of  sources  for  negotiations  [5:4-24].    The  concepts 
of  "significant  difference"  and  "essentially  technically  equal" 
will  be  discussed  later  in  this  chapter. 

The  TER  should  address  the  areas  of  the  remaining  pro- 
posals which  need  to  be  strengthened  to  correct  weaknesses  or 
areas  which  should  be  eliminated  or  reduced  to  correct  excesses 
Their  recommended  changes  should  be  accompanied  by  appropriate 
rationale  to  be  used  in  developing  the  negotiation  objectives. 

After  the  TER  is  written,  reviewed,  and  signed,  it  is 
forwarded  to  the  contracting  officer.   The  contracting  officer 
Is:  reaponsiible  for  reviewing  the  accuracy  and  completeness  of 
the  documentation.   This  review  provides  a  check  on  the 
reasioning  behind  the  justification  of  the  proposal  ratings  and 
a  check,  that  the  evaluation  was  conducted  in  accordance  with 
the  evaluation  factors  in  the  RFP.   Any  nonconforming  docu- 
mentation shall  be  returned  to  the  TEP  for  completeness,  cor- 
rection or  oth^er  remedial  action.   [20:13-7] 
3 .   Competitive  Range  Determination 

The  contracting  officer  is  also  responsible  for 
selecting  those  sources  with  whom  negotiations  will  be  con- 
ducted.  A  competitive  range  decision  is  based  on  cost/price, 
technical  and  other  salient  factors  and  includes  all  proposals 
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that  have  a  reasonable  chance  of  being  selected  for  award. 
A  competitive  range  means  that  two  or  more  proposals  are  accept- 
able for  negotiation  after  evaluation.   If  there  is  any  doubt 
as  to  whether  a  proposal  is  in  the  competitive  range,  it  must 
be  resolved  by  including  it.   A  summary  of  the  rules  pertaining 
to  competitive  range  determinations  is  listed  in  Chapter  IV. 
Those  sources  found  outside  the  competitive  range  are  labeled 
unacceptable  and  are  so  notified.   [5:4-26] 

G.   CONDUCTING  THE  NEGOTIATIONS 

1.   Development  of  Negotiation  Objectives 

The  Government  is  required  to  conduct  negotiations  with 
all  offerors  in  the  competitive  range  and  these  negotiations 
must  be  meaningful.   Normally,  negotiations  will  consist  of 
pointing  out  those  areas  in  which  the  proposal  is  deficient 
157]  ..   Th.e  common  problems  encountered  in  this  area  are  tech- 
nical leveling  and  technical  transfusion.   Technical  leveling 
is  the  improper  use  of  negotiations  resulting  in  multiple 
offerors  with,  approximately  the  same  technical  score.   Techni- 
cal transfusion  is  the  improper  transfer  of  innovative  ideas 
and  approaches  from  one  proposal  to  another.   Obviously, 
technical  transfusion  is  one  of  the  ways  to  bring  about  techni- 
cal leveling.   Another  common  way  is  to  concentrate  negotia- 
tion efforts  on  proposals  that  are  currently  unacceptable  but 
capable  of  being  made  acceptable.  For  example,  at  NRCO,  Long 
Beach  a  recent  protest  centered  on  an  offeror  that  was  able  to 
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improve  his  score  from  seventy-five  to  eighty-eight  due 
largely  to  deficiencies  pointed  out  during  negotiations.   An 
offeror  with  an  original  score  of  eighty-five  was  unable  to 
improve  his  proposal  and  protested  because  the  Government 
failed  to  point  out  deficiencies  known  in  his  already  accept- 
able, but  not  perfect  proposal  [11].   The  way  to  avoid  techni- 
cal leveling  is  to  point  out  all  significant  deficiencies  in 
all  proposals.   What  is  significant  must  be  determined  for 
each  individual  case  but  generally  means  any  deficiency  that 
may  affect  the  source  selection  results. 

Great  care  should  be  taken  by  all  Government  personnel 
to  prevent  technical  transfusion.   The  evaluation  of  each 
proposal  against  the  standards  set  forth  in  the  RFP  instead  of 
agains.t  each,  other  is  a  great  aid  in  this  effort.   Each  defici- 
enc3r  is  expressed  in  texms  of  how  it  fails  to  meet  a  standard 
or  is  in  great  excess  of  a  standard.   Deficiencies  are  not 
expressed  in  terms  of  how  another  proposal  is  superior. 
Evaluation  to  standards  assists  the  conversion  of  deficiencies 
to  negotiation  objectives  without  technical  transfusion.   If 
it  is  not  possible  to  discuss  an  inadequate  area  without  tech- 
nical transfusion,  then  negotiations  concerning  that  area  can- 
not be  held.   15  7] 

Another  consideration  is  later  requests  for  informa- 
tion under  the  Freedom  of  Information  Act  (FOIA).  TEP  members 
should  write  all  proposal  evaluations  recognizing  that  after 
awaxd  the  evaluation  report  may  be  provided  to  the  offeror 
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that  submitted  the  proposal.   This  may  result  in  technical 
transfusion  if  the  evaluation  report  contains  references  to 
other  proposals. 

The  selection  of  negotiation  objectives  will  not  be 
difficult  if  an  orderly  method  of  identification,  description 
and  recording  of  errors,  omissions  and  deficiencies  has  been 
followed  during  the  evaluation  process.   The  negotiation 
objective  may  also  address  scheduling,  cost/price  considera- 
tions, incentive  arrangements  or  special  provisions  to  be 
incorporated  into  the  contractual  document.   The  contracting 
officer  is  ultimately  responsible  for  selecting  the  negotia- 
tion objectives.   [16:3-26] 

2.   Strategies  and  Tactics 

Many  individuals  have  placed  a  lot  of  weight  on 
negotiation  strategies  and  tactics  but  adequate  preparations 
outweights.  any  maneuvers  used  during  negotiations.   The  con- 
tracting officer  will  represent  the  Government  in  any  negotia- 
tion and  will  determine  the  strategies  and  tactics  to  be  used. 
Each  contracting  officer  has  an  individual  style  for  conducting 
negotiation.   Therefore,  it  is  important  that  any  TEP  member 
asked  to  participate  fully  understands  how  the  contracting 
officer  intends  to  run  the  negotiations.   In  particular,  the 
TEP  member  must  understand  whether  he  has  a  free  rein  to  talk 
or  if  ha  must  wait  for  approval  from  the  contracting  officer 
before  engaging  in  discussions.   Much  will  depend  on  the 
atmosphere  of  the  negotiations.   [5:4-29] 
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Conducting  negotiations  with  several  offerors  in  a 
competitive  procurement  action  is  a  very  complex  process  with 
severe  restrictions  brought  about  by  the  requirements  for 
equal  treatment  of  offerors,  the  prevention  of  technical  trans- 
fusion and  the  prohibitions  against  auction  techniques.   This 
requires  a  more  structured  approach  to  the  preparation  for 
negotiations  and  a  more  disciplined  participation  by  TEP  mem- 
bers than  would  be  used  in  sole  source  negotiations.   The  cost 
and  fee  area,  which  is  the  primary  area  of  concern  in  a  sole 
source  procurement,  becomes  secondary  to  the  technical  nego- 
tiations.  This  is  largely  due  to  the  Government  assumption 
that  cost  and  fee  are  being  held  down  by  the  pressures  of  compe- 
tition and  the  greater  weight  assigned  to  technical  factors. 
15:4-30] 

3.   Bes:t  and  Final  CB&F) 

There  are  two  somewhat  unrelated  goals  of  the  negotia- 
tion prcceas.   Both,  the  technical  and  cost/price  evaluators 
are  attempting  to  gather  information  in  order  to  complete  the 
evaluation  process  and  the  contracting  personnel  are  attempting 
to  finalize  the.   terms  and  conditions  of  the  resultant  contract. 
The  evaluators  always  want  and  can  use  additional  information 
and  usually  are  forced  to  complete  the  evaluation  based  on 
incomplete  data.   At  the  end  of  the  negotiation  process,  the 
NFAS  activities  interviewed  request  revised  proposals  to  be 
received  by  a  common  cutoff  date.   Then  the  revised  proposals 
are  used  to  complete  the  evaluation  process.   Previously 

93 


unacceptable  offerors  that  were  capable  of  being  made  accept- 
able are  now  categorized  as  either  acceptable  or  not  accept- 
able.  A  final  competitive  range  determination  is  made.   The 
contracting  personnel  review  the  proposals  of  all  offerors  in 
the  competitive  range  to  ensure  there  are  no  unacceptable 
terms  and  conditions.   All  unsuccessful  offerors  are  notified. 
14] 

The  remaining  offerors  in  the  competitive  range  are 
then  given  one  final  opportunity  to  submit  revisions  which 
again  must  be  received  by  a  common  cutoff  date.   This  is  the 
Rest  and  Final  CB&F)  request  and  must  be  clearly  identified 
as  such-.   Offerors  have  a  right  to  know  that  negotiations  have 
concluded,  that  B&F  proposals  are  being  called  for  and  the 
common  cutoff  date  for  receipt.   [16:3-29] 

In  less  complex  requirements,  the  submission  of  revised 
proposals  and  the  request  for  B&F  proposals  are  compressed  into 
one  request  for  B&F  proposals.   However,  there  is  nothing  to 
prevent  an  offeror  from  suddenly  taking  exception  to  a  parti- 
cular terra  or  condition  or  revising  a  previously  acceptable 
proposal  in  a  way  that  makes  it  unacceptable.   These  problems 
may  not  be  able  to  be  resolved  without  negotiating  with  the 
offeror  but  if  negotiations  are  conducted  with  one  offeror, 
they  must  be  conducted  with  all  offerors.   Therefore,  it  may 
be  in  the  best  interest  of  the  Government  to  ask  for  another 
round  of  B&F  proposals.   This  leads  to  charges  of  auctioning 
and  delays  the  award,  but  is  permitted  where  justified  or 
necessary.   I4Q:7J 
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H.   FINAL  EVALUATIONS  AND  DOCUMENTATION 

The  final  evaluation  shall  take  into  account  the  results 
of  th.e  oral/written  negotiations  and  any  proposal  revisions 
submitted  by  offerors  as  a  part  of  their  B&F  proposals.   The 
offeror  may  respond  to  a  notification  of  a  weakness,  omission, 
or  deficiency  in  one  of  three  ways.   The  offeror  may  disagree 
that  a  deficiency  exists,  furnish  rebuttal  information  to 
substantiate  the  original  position,  and  stand  on  the  proposal 
as  submitted.   The  offeror  may  agree  that  a  deficiency  exists 
but,  because  of  previously  considered  trade-offs,  the  correc- 
tion of  the  deficiency  may  cause  loss  of  some  other  capability 
considered  to  be  more  valuable  and  determine  to  propose  with 
the  noted  deficiency.   Finally,  the  offeror  may  agree  that 
a  deficiency  exists  and  furnish  a  correction,   [14:47] 

The  final  evaluation  is  normally  conducted  as  an  update 
of  the  first  evaluation  and  not  a  totally  new  evaluation. 
Care  must  be  taken  to  evaluate  the  impact  of  revision  to  pro- 
posals not  only  in  the  area  of  the  original  deficiency  but  in 
all  areas  of  the  proposal.   The  final  documentation  is  in  the 
form  of  a  second  part  of  the  TER.   The  documentation  must 
clearly  reflect  the  effect  of  negotiations  and  explain  the 
basis  for  any  scoring  changes  that  were  made  [20:13-6].   The 
final  scores  and  documentation  must  reflect  the  integrated 
collective  judgment  of  the  TEP  as  to  the  competitive  standings 
of  the  offerors  overall  [12:4-13]. 
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I.   ULTIMATE  RESPONSIBILITY  FOR  SOURCE  SELECTION 

In  any  requirement  where  evaluation  factors  other  than 
price  are  used  and  award  is  based  on  the  technical  competency 
of  contractors,  the  primary  effort  in  source  selection  is  made 
by  the  TEP  members.   They  alone  are  presumed  capable  of  con- 
ducting the  review  of  proposals  and  recommending  and  documenting 
the.  award.   However,  the  ultimate  responsibility  for  choice  of 
a  contractor  remains  with  the  contracting  officer,  the  only 
person  who  can  legally  bind  the  Government  to  a  contract.   The 
law,  regulations  and  sensible  practices  all  require  that,  in 
the  last  analysis,  the  contracting  officer  must  weight  all  the 
evidence,  recommendations,  and  other  documentation  and  assume 
the  full  weight  of  the  ultimate  choice.   [2:51] 

Problems  usually  result  when  two  or  more  proposals  are 
evaluated  very  close  to  each  other.   The  proposals  may  be 
either  "essentially  technically  equal"  or  a  "significant 
difference"  may  exist.   The  contracting  officer  may  make  a 
formal  determination  backed  by  factual  data  that  the  proposals 
are  essentially  technically  equal.   This  is  in  recognition 
that  the  proposal  evaluation  is  sensitive  only  to  a  certain 
degree.   Of  the  proposals  determined  to  be  equal,  award  is  then 
made  to  the  lowest  priced  proposal  [59].   Whether  a  signifi- 
cant difference  exists  depends  on  the  facts  and  circumstances 
of  each  procurement  [61] . 

Depending  on  the  contract  type  and  dollar  value  of  the 
proposed  contract,  several  levels  of  review  may  be  required 

prior  to  award.   The  contracting  officer  may  try  to  expedite 
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the  process,  but  he  has  little  or  no  control  over  the 
reviewers.   The  time  for  the  reviews  is  subject  to  the  work- 
load of  the  reviewers.   [5:4-3] 

J.   UNSUCCESSFUL  OFFERORS 

1.  Notification 

There  are  three  basic  reasons  why  and  three  basic 
times  when  an  offeror  may  be  notified  that  his  proposal  is 
unsuccessful.   A  proposal  may  be  unacceptable,  acceptable 
but  outside  the  competitive  range  or  acceptable  but  simply 
not  selected  for  award.   The  offeror  must  be  promptly  noti- 
fied that  his  proposal  is  no  longer  being  considered  for 
contract  award.   A  notification  stating  that  the  proposal  was 
unacceptable  shall  cite  the  appropriate  reasons.   [12:4-15] 

2.  Debr  ief  Ing 

The  basic  purpose  of  a  debriefing  is  to  furnish 
inforiaatlon  that  should  permit  the  offeror  to  submit  a  better 
proposal  in  the  future.   Government  personnel  may  also  learn 
valuable  information,  th.erefore,  a  debriefing  is  often  in  the 
best  interests  of  the  Government.   Any  debriefing  is  to  be 
conducted  by  tha  contracting  officer  with  necessary  support 
provided  by  the  TEP  members.   The  debrief ings  are  not  to  be 
conducted  until  after  contract  award  and  with  only  one  offeror 
at  a  time.   The  debriefing  generally  consists  of  a  general 
discussion  of  the;  offeror's  proposal,  its  weak  and  strong 
points  in  relation  to  the  requirements  of  the  RFP  and  not 
relative  to  the  other  proposals  [14:66].   It  is  Imperative 
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that  the  TEP  members  be  cautioned  not  to  discuss  other  pro- 
posals and  to  refrain  from  implying  that  the  evaluation  was 
conducted  using  any  factors  other  than  those  set  forth  in  the 
RFP  [54] . 

Th.e  offerors  should  be  encouraged  to  submit  written 
questions,  to  the  Government  prior  to  the  actual  debriefing. 
This  will  allow  a  much  more  efficient  use  of  time  during  the 
debriefing.   Of  utmost  importance  in  debriefing  discussions 
is  the  use  of  tact.   The  TEP  member  should  answer  appropriate 
questions  pertaining  to  the  defects  found  in  the  proposal  in 
a  straightforward  manner.   At  the  conclusion  of  the  debriefing, 
the  contracting  officer  should  develop  a  record  of  the  de- 
briefing including  the  questions  asked,  the  answers  given  and 
ainy  other  apporpriate  data.   [14:67] 
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VII.   SUMMARY  AND  RECOMMENDATIONS 

A.   SUMMARY 

As  an  aid  to  the  technical  evaluator,  a  model  of  the 
major  phases  and  significant  events  in  the  proposal  evalua- 
tion and  source  selection  process  is  provided  as  figure  10. 
Various  administrative  and  review  events  internal  to  the 
contracting  activity  are  not  included.   Individual  procure- 
ment actions  may  have  minor  variations  from  the  model.   For 
e:xample,  award  on  initial  offers  would  eliminate  several 
significant  events. 

Although  it  is  obviously  better  for  a  technical  evaluator 
to  become  involved  in  a  particular  acquisition  as  early  as 
possible,  he  may  enter  th.e  process  at  any  one  of  the  five 
phases.  Most  often,  they  enter  in  the  early  stages  of  the 
pre—solicitation  phase.   It  is  imperative  that  the  technical 
evaluator  understand  what  phase  he  is  in  and  the  resulting 
constraints  with  each  phase. 

1.   phase  I  Pre-procurement  Request 

Th.e  importance  of  an  informed  start  cannot  be  over 
emphasized.   Initially,  technical  personnel  should  be  encour- 
aged to  describe  th.e  desired  services  free  of  constraints  in 
order  to  promote  innovation  and  the  identification  of  needs . 
Open  communication  with,  private  industry  and  contracting 
personnel  is  important  in  order  to  gather  the  necessary 
information.   However,  as  the  requirement  starts  to  take  shape, 
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communication  with  private  industry  becomes  more  constrained 
in  order  not  to  give  a  competitive  advantage  to  any  particular 
contractor. 

As  soon  as  a  significant  requirement  for  services  is 
firm  enough,  to  provide  a  basic  description,  contracting 
personnel  should  be  contacted  to  discuss  the  potential  for 
competition,  type  of  contract,  and  source  selection  method. 
If  the  source  selection  method  chosen  requires  the  use  of 
evaluation  factors,  contracting  personnel  should  fully  explain 
their  proper  use.   The  statement  of  work  (SOW)  should  be 
written  with  standards  that  consistently  establish  a  particular 
quality  level.   Evaluation  factors  selected  must  be  consistent 
with,  the  SOW  and  the  established  standards.   The  procurement 
request  CPR)  must  also  include  a  budget  estimate  of  the  funds 
required. 

2.   Phase  II  Pre-solicitation 

The  pre-solicitation  phase  starts  with  the  receipt  of 
the  PR  by  the  NFAS  activity  and  ends  when  the  Request  for 
Proposal  CPFP)  is  issued.   This  phase  is  critical  for  success- 
ful sraurce  selection  because  of  its  impact  on  future  events 
and  activities.   Sufficient  time  should  be  provided  for 
planning  and  preparation.   The  contract  type  is  a  major  con- 
sideration of  potential  offerors.   The  selection  of  competent 
and  qualified  Technical  Evaluation  Panel  (TEP)  members  influ- 
ences the  quality  of  the  evaluations  and  ultimate  source  selec- 
tion.  The  SOW  and  evaluation  factors  selected  impact  the 
ability  of  potential  offerors  to  submit  proposals  that  meet  the 
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needs  of  the  Government.   The  Source  Selection  Plan  (SSP) , 
which  includes  evaluation  factors,  weights,  scoring  system, 
and  proposed  event  milestones,  impacts  the  smoothness  and 
objectivity  of  the  source  selection.   The  issuance  of  the  RFP 
sets  the  terins  and  conditions  of  the  source  selection  process 
for  each  procurement  action,  which  cannot  be  changed  unless 
the  RFP  is  revised.   [73-6] 

3,  Phase  III  Solicitation 

The  solicitation  phase  is  normally  the  most  stable  of 
the  five  phases .   However ,  the  pre-proposal  conference  and 
other  communication  with  contractors  provide  an  opportunity  to 
prevent  potential  problems  that  may  arise  in  later  phases. 
The  solicitation  document  may  require  amendment  or  the  SSP  may 
need  to  be  revised. 

4.  Phase  IV  Evaluation 

The  evaluation  phase  begins  with  the  receipt  of  pro- 
posals but  does  not  have  a  natural  ending.   Evaluations  may 
continue  right,  up  until  the  award  decision  is  finalized. 
Grossly  deficient  or  clearly  inferior  proposals  may  be  elimi- 
nated at  any  time  from  furtiier  consideration.   The  evaluation 
phase  has  a  dual  purpose  of  ranking  the  proposals  received 
and  determixiing  appropriate  negotiation  objectives.   A  com- 
plete record  is  maintained  and  made  a  part  of  the  contract 
file. 
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5 .   Phase  V  Source  Selection 

The  source  selection  phase  starts  with  the  completion 
of  the  initial  proposal  evaluation.   Several  important  decisions 
are.  necessary  in  the  sequence  leading  up  to  the  final  source 
selection  decision.   The  competitive  range  determination  is  a 
decision  to  limit  further  negotiations  to  those  proposals  con- 
sidered to  stand  a  reasonable  chance  of  subsequent  contract 
award.   Another  major  decision  is  the  extent  to  which  written 
or  oral  negotiations  should  be  conducted  to  give  the  offerors 
the  opportunity  to  clarify,  correct  or  improve  their  respective 
proposals.   These  decisions  greatly  affect  the  final  source 
selection  decision  and  provide  fertile  areas  for  contractor 
dissatisfaction  and  protest. 

B .   RECQMMENDAT IONS 

1.   Simple  Multi-attribute  Rating  Technique  (SMART) 

It  is  recommended  that  the  SMART  method  of  evaluation 
be  reviewed  for  adoption  by  the  Naval  Supply  Systems  Command. 
Because  SMART  is  specifically  designed  for  the  non-professional 
analyst,  it  is  uniquely  suited  to  proposal  evaluation  at  NFAS 
activities.   Most  NFAS  activities  currently  use  a  similar 
system  and  could  easily  adapt  to  SMART.   A  single,  well  docu- 
mented technique  such  as  SMART  would  improve  the  quality  of 
evaluations  and  provide  consistency  within  and  between 
activities. 
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2.  Source  Selection  Plan  (SSP) 

It  is  recommended  that  NFAS  activities  be  encouraged 
to  use  formal  Source  Selection  Plans  (SSP)  with  established 
event  milestones  when  evaluation  factors  are  being  used.   This 
would  encourage  a  greater  effort  in  the  planning  states  and 
aid  in  the  administration  of  the  proposal  evaluation  and 
source  selection  process.   It  would  also  promote  coordination 
between  the  statement  of  work  and  the  evaluation  factors. 

3 .  Summary  of  Rules 

It  is  recommended  that  a  more  complete  summary  of  the 
rules  pertaining  to  proposal  evaluation,  source  selection, 
and  service  contracting  be  prepared  and  maintained  by  the 
Office  of  Legal  Counsel,  Naval  Supply  Systems  Command.   The 
potential  for  protest  can  be  reduced  by  appropriate  planning 
If  predecents  set  by  GAO  are  known  to  field  technical  and 
contracting  personnel.   Preparation  and  review  by  counsel  is 
necessary  to  provide  validity  and  to  keep  the  summary  current. 

4.  Guide  for  Technical  Personnel 

It  is  recommended  that  the  Naval  Material  Command  issue 
a  guide  for  technical  personnel  in  order  to  assist  them  in 
their  duties  as  proposal  evaluators.   A  need  for  guidance  and 
training  was  universally  recognized  by  all  individuals  inter- 
viewed at  NFAS  activities.   A  guide  was  considered  more  desir- 
able than  a  training  course  because  it  can  be  more  widely  dis- 
tributed.  This  guide  should  contain  essentially  the  same 
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information  as  contained  in  this  research  effort,  but  in  a 
compressed,  easy  to  read  form. 


I 
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